Deep Fried Convnets

The fully connected layers of a deep convolutional neural network typically contain over 90% of the network parameters, and consume the majority of the memory required to store the network parameters. Reducing the number of parameters while preserving essentially the same predictive performance is critically important for operating deep neural networks in memory constrained environments such as GPUs or embedded devices. In this paper we show how kernel methods, in particular a single Fastfood layer, can be used to replace all fully connected layers in a deep convolutional neural network. This novel Fastfood layer is also end-to-end trainable in conjunction with convolutional layers, allowing us to combine them into a new architecture, named deep fried convolutional networks, which substantially reduces the memory footprint of convolutional networks trained on MNIST and ImageNet with no drop in predictive performance.

💡 Research Summary

The paper addresses the well‑known problem that fully‑connected (FC) layers in deep convolutional neural networks (CNNs) account for more than 90 % of the total parameters, making memory consumption a bottleneck for training and deployment on GPUs or embedded devices. To alleviate this, the authors propose replacing every FC layer with an Adaptive Fastfood transform, a trainable extension of the Fastfood random‑projection technique originally introduced for kernel approximation.

In the classic Fastfood construction a linear mapping is expressed as y = SHGΠHBx, where S, G, and B are diagonal matrices, H is the Walsh‑Hadamard matrix, and Π is a random permutation. The original method fixes S, G, B as random constants; the Adaptive version instead treats them as learnable parameters that are updated by standard back‑propagation. This change turns the transform into a flexible, data‑dependent feature extractor while preserving the computational advantages of the original scheme.

The authors derive the storage and computational complexities of the Adaptive Fastfood layer. For an input of dimension d and an output of dimension n, the storage requirement drops from O(nd) (the size of a dense weight matrix) to O(n), because only the diagonal entries of S, G, B and the permutation indices need to be stored. The forward pass costs O(n log d) operations, thanks to the fast Hadamard transform, and the backward pass can be computed with the same asymptotic cost. When n > d, the layer can be tiled n/d times, yielding overall costs of O(n log d) for computation and O(n) for storage—substantial savings compared with the O(nd) baseline.

Two complementary theoretical viewpoints are offered. From the structured random‑projection perspective, the Adaptive Fastfood transform can be seen as a Johnson‑Lindenstrauss‑type embedding whose projection matrix is highly compressible yet still preserves distances in expectation. By learning the diagonal scalings, the embedding adapts to the data distribution, overcoming the variance issues of naïve sparse random matrices. From the kernel‑approximation perspective, Fastfood implements Monte‑Carlo approximations of shift‑invariant kernels (e.g., RBF) by replacing the dense random matrix Wx in random‑feature maps with SHGΠHBx. The learnable diagonals correspond to adjusting the spectral density (bandwidth), scaling, and even the kernel family itself, effectively learning a data‑driven kernel within the network.

Empirically, the method is evaluated on MNIST and ImageNet. On MNIST, a standard CNN with two FC layers is swapped for Adaptive Fastfood layers, achieving the same >99 % test accuracy while reducing the total number of parameters by roughly 55 %. On ImageNet, a VGG‑like architecture is modified in the same way; the resulting “deep fried” network attains the same Top‑1 accuracy (≈71.5 %) as the baseline while using only about half the parameters. The authors also experiment with applying the transform to the final softmax classifier, confirming that performance does not degrade. Memory usage on a single GPU is cut dramatically, enabling training of the full ImageNet model on hardware that would otherwise be insufficient.

The paper situates its contribution among prior model‑compression techniques such as low‑rank factorization, pruning, hashing tricks, and knowledge distillation. Unlike pruning or hashing, Adaptive Fastfood does not require a full‑size model to be trained first; it is a lightweight design from the outset. Compared with low‑rank factorization, it reduces both storage and compute, because the Fastfood structure yields O(n log d) multiplication cost rather than O(nd) even after factorization. Knowledge distillation remains a post‑hoc approach, whereas Adaptive Fastfood is integrated directly into the training pipeline.

In conclusion, the Adaptive Fastfood transform offers a principled, end‑to‑end trainable mechanism to compress fully‑connected layers, delivering substantial memory savings and computational speed‑ups without sacrificing predictive performance. Its dual interpretation as a structured random projection and as a learnable kernel approximation opens avenues for further research, such as combining it with other compression schemes, extending it to recurrent architectures, or exploring alternative diagonal parameterizations. The work demonstrates that “deep fried” networks are a practical solution for deploying high‑accuracy CNNs in memory‑constrained environments.