Robust Representation Learning in Masked Autoencoders

Masked Autoencoders (MAEs) achieve impressive performance in image classification tasks, yet the internal representations they learn remain less understood. This work started as an attempt to understand the strong downstream classification performance of MAE. In this process we discover that representations learned with the pretraining and fine-tuning, are quite robust - demonstrating a good classification performance in the presence of degradations, such as blur and occlusions. Through layer-wise analysis of token embeddings, we show that pretrained MAE progressively constructs its latent space in a class-aware manner across network depth: embeddings from different classes lie in subspaces that become increasingly separable. We further observe that MAE exhibits early and persistent global attention across encoder layers, in contrast to standard Vision Transformers (ViTs). To quantify feature robustness, we introduce two sensitivity indicators: directional alignment between clean and perturbed embeddings, and head-wise retention of active features under degradations. These studies help establish the robust classification performance of MAEs.

💡 Research Summary

This paper investigates why Masked Autoencoders (MAEs) achieve strong downstream classification performance and how their internal representations behave under image degradations. The authors first conduct a layer‑wise analysis of the pretrained MAE encoder (ViT‑Base, 12 layers, 12 heads per layer). For each layer they extract three types of token embeddings: the CLS token, individual visible‑patch tokens, and the mean‑patch embedding (the average of all visible patches). t‑SNE visualizations show that early layers contain little class structure, while deeper layers gradually form distinct clusters for each class. To move beyond qualitative plots, the authors perform a subspace‑based geometric study: for every class they collect all patch embeddings at a given layer, apply singular value decomposition, and keep the top‑k singular vectors as a low‑dimensional subspace. Principal angles between class‑specific subspaces are then measured across layers. The angles are small in the first few layers (subspaces overlap) but increase sharply after layer 8, indicating that the encoder progressively builds class‑separable subspaces even though it is trained without labels.

Next, the paper evaluates the robustness of a fine‑tuned MAE on ImageNet‑1k under two controlled perturbations: Gaussian blur (varying kernel size and σ to produce a monotonic decrease in PSNR/SSIM) and attention‑guided occlusion (using attention rollout to mask the most attended patches). Classification accuracy remains high across a wide range of blur levels and degrades gracefully under occlusion, consistently outperforming a standard Vision Transformer (ViT) by 5–7 % in the occlusion scenario.

To quantify representation robustness, two complementary indicators are introduced. (1) Directional alignment measures the cosine similarity between the mean‑patch embedding of a clean image and that of its perturbed version. Across both blur and occlusion, the similarity stays above 0.9, showing that the encoder projects degraded inputs into nearly the same direction in latent space. (2) Head‑wise active‑feature retention examines each attention head’s value matrix. Features with large magnitude are deemed “active.” The authors compute the overlap of active‑feature sets between clean and perturbed inputs for every head and layer. Deeper layers retain over 80 % of active features, indicating that high‑level representations are especially stable.

The combined evidence leads to several key insights: (i) MAE’s asymmetric encoder–decoder design forces the encoder to learn global, class‑aware structures despite extreme masking; (ii) these structures emerge as low‑dimensional subspaces that diverge with depth, providing a strong initialization for downstream fine‑tuning; (iii) the learned latent space is inherently robust to common degradations, which explains the observed stability in classification performance; and (iv) the proposed robustness metrics can serve as diagnostic tools for future self‑supervised vision models.

Overall, the work deepens our understanding of MAE beyond reconstruction loss, revealing that its success stems from an emergent, geometrically separable latent space that remains resilient under realistic image perturbations. This insight has practical implications for designing more robust self‑supervised vision systems and for evaluating robustness in other masked‑modeling frameworks.