ViT-5: Vision Transformers for The Mid-2020s

This work presents a systematic investigation into modernizing Vision Transformer backbones by leveraging architectural advancements from the past five years. While preserving the canonical Attention-FFN structure, we conduct a component-wise refinement involving normalization, activation functions, positional encoding, gating mechanisms, and learnable tokens. These updates form a new generation of Vision Transformers, which we call ViT-5. Extensive experiments demonstrate that ViT-5 consistently outperforms state-of-the-art plain Vision Transformers across both understanding and generation benchmarks. On ImageNet-1k classification, ViT-5-Base reaches 84.2% top-1 accuracy under comparable compute, exceeding DeiT-III-Base at 83.8%. ViT-5 also serves as a stronger backbone for generative modeling: when plugged into an SiT diffusion framework, it achieves 1.84 FID versus 2.06 with a vanilla ViT backbone. Beyond headline metrics, ViT-5 exhibits improved representation learning and favorable spatial reasoning behavior, and transfers reliably across tasks. With a design aligned with contemporary foundation-model practices, ViT-5 offers a simple drop-in upgrade over vanilla ViT for mid-2020s vision backbones.

💡 Research Summary

The paper introduces ViT‑5, a systematic modernization of the classic Vision Transformer (ViT) architecture that incorporates a suite of component‑wise refinements proven effective in large language models over the past five years. The authors deliberately retain the canonical Attention‑FFN backbone, focusing instead on updating normalization, activation scaling, positional encoding, gating mechanisms, and learnable register tokens.

Key architectural updates

1. LayerScale – a learnable per‑channel scaling factor applied to each block’s output. The authors show mathematically that LayerScale is equivalent to a post‑RMSNorm scaling, but offers lower overhead and greater flexibility, thus becoming the default scaling mechanism.
2. RMSNorm – replaces LayerNorm throughout the model. By removing the centering operation, RMSNorm reduces unnecessary shift noise, slightly lowers compute, and yields a modest accuracy gain (+0.2 % top‑1 on ImageNet‑1k for the base size).
3. Activation function – although modern LLMs often use SwiGLU, ViT‑5 retains GeLU. Experiments reveal that combining LayerScale with SwiGLU leads to “over‑gating”: both act as channel‑wise filters, producing overly sparse intermediate activations and degrading performance. Hence the original GeLU‑based MLP is kept.
4. Positional encoding – ViT‑5 jointly employs absolute positional embeddings (APE) and 2‑D rotary positional embeddings (RoPE). RoPE supplies relative distance information and improves resolution robustness, while APE preserves absolute spatial cues that are essential for many vision tasks. The authors demonstrate that a RoPE‑only design makes the model invariant to patch‑level flips, which is undesirable for generic backbones.
5. Register tokens – additional learnable tokens appended to the patch sequence. When combined with high‑frequency RoPE, these tokens mitigate activation artifacts and enhance long‑range token interactions, leading to better spatial reasoning.

Empirical results

• Image classification: ViT‑5‑Base achieves 84.2 % top‑1 accuracy on ImageNet‑1k under comparable FLOPs, surpassing DeiT‑III‑Base (83.8 %). Similar gains are observed for small and large variants.
• Generative modeling: Integrated as the visual encoder in a SiT diffusion pipeline, ViT‑5 reduces FID from 2.06 (vanilla ViT) to 1.84, a ~10 % improvement.
• Dynamic resolution robustness: Trained at 224 × 224, ViT‑5 maintains or improves accuracy across test resolutions from 128 to 512 px, whereas DeiT‑III’s performance drops sharply away from the training size.
• Spatial reasoning: Attention visualizations reveal clearer, more localized activation patterns in ViT‑5, confirming that the combined APE + RoPE + register token design enhances spatial understanding.

Ablation insights
The study shows that not all modern components are orthogonal. Naïvely stacking every recent improvement can hurt performance; for example, LayerScale + SwiGLU leads to over‑gating, and RoPE alone introduces flip invariance. Careful selection and interaction design are therefore crucial.

Conclusion
ViT‑5 demonstrates that the ViT backbone, despite its simplicity, remains under‑optimized. By systematically integrating RMSNorm, LayerScale, dual positional encodings, and register tokens—while avoiding incompatible gating—ViT‑5 delivers consistent gains across classification, generation, and dense prediction tasks without increasing computational budget. The work narrows the architectural gap between vision and language transformers and offers a drop‑in, future‑proof upgrade for mid‑2020s vision models.