Revisit Visual Prompt Tuning: The Expressiveness of Prompt Experts

Visual Prompt Tuning (VPT) has proven effective for parameter-efficient adaptation of pre-trained vision models to downstream tasks by inserting task-specific learnable prompt tokens. Despite its empirical success, a comprehensive theoretical understanding of VPT remains an active area of research. Building on the recently established connection between Mixture of Experts (MoE) and prompt-based methods, wherein each attention head can be conceptualized as a composition of multiple MoE models, we reinterpret VPT as the introduction of new prompt experts into these MoE structures. We identify a key limitation in existing VPT frameworks: the restricted functional expressiveness of prompt experts, which remain static and thus limited in their adaptability. To address this, we propose Visual Adaptive Prompt Tuning (VAPT), a novel method that endows prompt experts with enhanced expressiveness while preserving parameter efficiency. Empirical evaluations on VTAB-1K and FGVC demonstrate that VAPT achieves substantial performance improvements, surpassing fully fine-tuned baselines by 7.34% and 1.04%, respectively. Moreover, VAPT consistently outperforms VPT while requiring fewer additional parameters. Furthermore, our theoretical analysis indicates that VAPT achieves optimal sample efficiency. Collectively, these results underscore the theoretical grounding and empirical advantages of our approach.

💡 Research Summary

The paper revisits Visual Prompt Tuning (VPT), a parameter‑efficient method for adapting large pre‑trained Vision Transformers (ViTs) to downstream tasks by inserting learnable prompt tokens. While VPT has demonstrated strong empirical results, its theoretical foundations remain under‑explored. Recent work has shown that each multi‑head self‑attention (MSA) layer in a transformer can be interpreted as a composition of multiple Mixture‑of‑Experts (MoE) models. Within this MoE view, the pre‑trained linear experts correspond to the original token embeddings, whereas the prompt tokens act as additional “prompt experts” that are added to the MoE mixture.

The authors identify a critical limitation: the prompt experts in standard VPT are static vectors (p₁,…,p_Np) that do not depend on the input. Although the gating scores for these experts are input‑dependent, the experts themselves are constant functions, which contrasts with the input‑adaptive nature of typical MoE experts and reduces the expressive power of the prompt. Prior theoretical analyses (e.g., Petrov et al., 2023) suggest that static prompts can only contribute a bias term, further confirming this bottleneck.

To overcome this, the paper proposes Visual Adaptive Prompt Tuning (VAPT). VAPT introduces two lightweight modules that endow prompt experts with input‑dependent functionality while keeping the overall parameter budget low:

1. Token‑wise Projectors: For each prompt token j′, a learnable scalar weight α_{j′,k} is assigned to every input token k. The projector computes a weighted sum G_{j′}(X)=∑{k=1}^N α{j′,k} x_k, where X=