Blocks and Fuel: Frameworks for deep learning

We introduce two Python frameworks to train neural networks on large datasets: Blocks and Fuel. Blocks is based on Theano, a linear algebra compiler with CUDA-support. It facilitates the training of complex neural network models by providing parametrized Theano operations, attaching metadata to Theano’s symbolic computational graph, and providing an extensive set of utilities to assist training the networks, e.g. training algorithms, logging, monitoring, visualization, and serialization. Fuel provides a standard format for machine learning datasets. It allows the user to easily iterate over large datasets, performing many types of pre-processing on the fly.

💡 Research Summary

This paper presents two Python libraries—Blocks and Fuel—designed to streamline the development, training, and data handling of deep neural networks built on top of Theano. Blocks focuses on model construction and training management, while Fuel provides a unified interface for large‑scale datasets and on‑the‑fly preprocessing.

Blocks introduces the concept of “bricks,” which are parametrized Theano operations encapsulating shared variables (weights, biases, filters, etc.) and optional sub‑bricks. By annotating Theano’s symbolic graph with metadata tags (e.g., role = weight, bias, filter), Blocks enables fine‑grained queries and transformations without abstracting away Theano’s native graph. This approach preserves Theano’s automatic differentiation, GPU acceleration, and graph optimizations while offering a hierarchical, object‑oriented view of complex models such as stacked LSTMs, GRUs, and attention mechanisms. Bricks can be initialized using a variety of schemes (sparse, orthogonal, etc.) and can be combined to form deep feed‑forward, convolutional, or recurrent architectures.

Graph management utilities allow generic regularization (weight decay, dropout, weight noise, gradient clipping) to be applied based on tags rather than hard‑coded layer names. A query language lets users target specific subsets of parameters—for example, adding Gaussian noise only to the recurrent weights of LSTM units whose parent brick is named “encoder.” This model‑agnostic regularization is especially valuable for research experiments that frequently modify network components.

Training in Blocks is organized around a “main loop” that couples a Theano computation graph, a Fuel data stream, and a set of step‑rules. Step‑rules implement learning‑rate scaling, momentum, AdaGrad, Adam, RMSProp, etc., and can be stacked or swapped at runtime. The main loop also supports extensible callbacks for monitoring validation performance, logging, plotting, checkpointing, and learning‑rate scheduling. Checkpointing is made robust by re‑implementing Python’s iter‑tools module to be pickle‑compatible, allowing experiments to be paused, serialized, and resumed without loss of reproducibility.

Fuel addresses the often‑overlooked data‑pipeline side of deep learning. It defines a standard HDF5‑based storage format that consolidates all data sources (features, targets, masks) and associated metadata (axis semantics, split definitions, availability) into a single file. This format is efficient for out‑of‑core datasets and simplifies random access. Fuel provides a flexible iteration API that supports sequential or shuffled minibatches, in‑memory or out‑of‑core loading, and resampling strategies such as cross‑validation or bootstrapping. Preprocessing steps (random cropping, n‑gram generation, data augmentation, etc.) are expressed as composable pipelines executed in a separate process to avoid the Global Interpreter Lock (GIL). Processed batches are transferred to the training process via TCP sockets, ensuring that the main training loop remains CPU‑bound only on the model computation.

Fuel also supplies command‑line utilities for downloading popular public datasets (MNIST, CIFAR‑10, ImageNet) and converting them to the standardized HDF5 format. The conversion script automatically records the originating module version, command used, and timestamps as metadata, facilitating exact reconstruction of the dataset preparation pipeline.

Both libraries emphasize software engineering best practices: high test coverage, thorough documentation, tutorials, and active mailing lists. Example repositories demonstrate state‑of‑the‑art models such as neural machine translation (Bahdanau et al., 2015) and the Deep Recurrent Attentive Writer (Gregor et al., 2015) implemented with Blocks, showcasing the frameworks’ capability to handle sophisticated architectures.

In summary, Blocks and Fuel together provide a cohesive ecosystem for Theano‑based deep learning research. Blocks offers a metadata‑driven, hierarchical abstraction over Theano graphs, enabling flexible model definition, generic regularization, and modular training loops. Fuel standardizes dataset storage, iteration, and preprocessing, delivering scalable data pipelines that integrate seamlessly with Blocks. Their design choices prioritize rapid prototyping, reproducibility, and extensibility, making them valuable tools for researchers developing and evaluating complex neural network models.