These Are Not All the Features You Are Looking For: A Fundamental Bottleneck in Supervised Pretraining

Transfer learning is widely used to adapt large pretrained models to new tasks with only a small amount of new data. However, a challenge persists – the features from the original task often do not fully cover what is needed for unseen data, especially when the relatedness of tasks is not clear. Since deep learning models tend to learn very sparse representations, they retain only the minimal features required for the initial training while discarding potentially ones for downstream transfer. A theoretical framework developed in this work demonstrates that such pretraining captures inconsistent aspects of the data distribution, therefore, inducing transfer bias. To address this limitation, we propose an inexpensive ensembling strategy that aggregates multiple models to generate richer feature representations. On ResNet, this approach yields a $9%$ improvement in transfer accuracy without incurring extra pretraining cost. We also present empirical evidence from a range of deep learning studies, confirming that the phenomenon is pervasive across modern deep learning architectures. These results suggests that relying solely on large pretrained networks is not always the most effective way to improve model generalization. Instead, fostering richer, more diverse representations – e.g. - through model ensembles – can substantially enhance transfer learning performance.

💡 Research Summary

This paper investigates a fundamental limitation of supervised pre‑training for transfer learning: the features learned on a large, mixed source dataset often fail to cover all the information needed for downstream tasks, even when the target task is explicitly present in the pre‑training mixture. The authors attribute this shortfall to a “sparsity bias” inherent in modern deep networks trained with stochastic gradient descent, large learning rates, early stopping, and weight decay. Under this bias, the network tends to retain only the minimal set of features that reduce training loss on the source mixture, discarding other potentially useful features that could be crucial for later adaptation.

The theoretical framework formalizes the problem as follows. Let {P⁽ʲ⁾} be a collection of data distributions and P_mix = Σ_j λ_j P⁽ʲ⁾ a weighted mixture used for pre‑training. For a target distribution P⁽ᵢ⁾, the paper compares two regimes: (1) direct training on P⁽ᵢ⁾, learning both feature extractor θ_i and classifier γ_i; and (2) transfer learning where the extractor θ_mix is frozen after pre‑training and only a linear classifier γ_i^lin is learned on the small target set (linear probing). The central question is whether the function space spanned by the frozen features {γ^T φ(·;θ_mix)} contains the optimal solutions obtained by direct training. The authors show that, under realistic assumptions (unlimited source data but non‑zero training error, limited target data, and linear probing), the answer is generally negative.

A key insight is that fine‑tuning in this regime can be interpreted as a first‑order Taylor expansion around the pre‑trained weights, which reduces to learning a linear combination of Neural Tangent Kernel (NTK) features. Consequently, if the pre‑trained feature space lacks certain dimensions, fine‑tuning cannot create them; it can only re‑weight existing directions. This leads to a concrete counterexample: a toy feature space with two basis functions φ₁ and φ₂ and four sub‑distributions, each requiring only one of the two features for perfect classification. The mixture of these sub‑distributions is not linearly separable; the optimal classifier must ignore the least‑weighted point and can be represented by a single feature. Because of sparsity bias, a network trained on the mixture will select either φ₁ or φ₂, thereby correctly handling only two of the four sub‑tasks. This demonstrates that even when the target task is present in the pre‑training data, essential features may be omitted.

Empirically, the authors validate the theory on ResNet‑50 and several other architectures. They pre‑train multiple instances of the same architecture with different random seeds and data shuffling orders, then evaluate transfer performance on downstream benchmarks (e.g., MNIST‑Fashion, CIFAR‑10). An inexpensive ensembling strategy—averaging logits or voting across the independently trained models—produces richer feature representations without any extra pre‑training cost. Across experiments, ensembling yields an average 9 % absolute gain in transfer accuracy (e.g., from 81 % to 90 % on CIFAR‑10, from 94 % to 97 % on MNIST‑Fashion). The paper also surveys a broad set of recent deep‑learning studies (including large foundation models in genomics) and finds consistent evidence of the sparsity‑induced feature loss, reinforcing the claim that the bottleneck is pervasive.

The paper’s contributions can be summarized as follows:

1. It challenges the prevailing assumption that massive pre‑training automatically captures “all useful features” for any downstream task.
2. It formalizes a sparsity bias mechanism that explains why certain features are systematically omitted during pre‑training.
3. It provides a simple, low‑cost remedy—model ensembling—that enriches the feature space and substantially improves transfer performance.
4. It demonstrates the generality of the phenomenon across vision, language, and scientific domains.

In conclusion, the work argues that relying solely on ever larger, all‑purpose foundation models is not a guaranteed path to optimal transfer learning. Instead, encouraging feature diversity—through inexpensive ensembles or other mechanisms that counteract sparsity bias—offers a more reliable route to robust generalization on unseen tasks.