Visual Autoregressive Modeling for Instruction-Guided Image Editing

Recent advances in diffusion models have brought remarkable visual fidelity to instruction-guided image editing. However, their global denoising process inherently entangles the edited region with the entire image context, leading to unintended spurious modifications and compromised adherence to editing instructions. In contrast, autoregressive models offer a distinct paradigm by formulating image synthesis as a sequential process over discrete visual tokens. Their causal and compositional mechanism naturally circumvents the adherence challenges of diffusion-based methods. In this paper, we present VAREdit, a visual autoregressive (VAR) framework that reframes image editing as a next-scale prediction problem. Conditioned on source image features and text instructions, VAREdit generates multi-scale target features to achieve precise edits. A core challenge in this paradigm is how to effectively condition the source image tokens. We observe that finest-scale source features cannot effectively guide the prediction of coarser target features. To bridge this gap, we introduce a Scale-Aligned Reference (SAR) module, which injects scale-matched conditioning information into the first self-attention layer. VAREdit demonstrates significant advancements in both editing adherence and efficiency. On EMU-Edit and PIE-Bench benchmarks, VAREdit outperforms leading diffusion-based methods by a substantial margin in terms of both CLIP and GPT scores. Moreover, VAREdit completes a 512$\times$512 editing in 1.2 seconds, making it 2.2$\times$ faster than the similarly sized UltraEdit. Code is available at: https://github.com/HiDream-ai/VAREdit.

💡 Research Summary

The paper introduces VAREdit, a novel framework for instruction‑guided image editing that leverages visual autoregressive (VAR) modeling instead of the dominant diffusion‑based approaches. Diffusion models achieve impressive visual fidelity through iterative denoising, but their global denoising process inevitably entangles edited regions with the surrounding context, causing unintended modifications and incurring high computational cost. VAR models, by contrast, treat image synthesis as a sequential token‑by‑token generation task over discrete visual tokens, naturally preserving unchanged areas while allowing precise modifications.

VAREdit reframes editing as a “next‑scale prediction” problem. An image is first tokenized by a multi‑scale visual tokenizer that produces a hierarchy of residual maps (R_1 … R_K) (coarse‑to‑fine). A transformer then autoregressively predicts these residuals one scale at a time. At each step (k), the model aggregates all previously generated residuals into a cumulative feature map (F_k), downsamples it to match the spatial resolution of the next scale, and feeds the resulting embedding (eF_k) into the transformer to predict the next residual (R_{k+1}). After all (K) residuals are generated, they are upsampled and summed to reconstruct the final feature map, which a decoder transforms into the edited image.

The central technical challenge is how to condition the generation on the source image efficiently. Conditioning on the full set of source residuals (F_{src}^{1:K}) provides complete information but doubles the token sequence length, leading to quadratic self‑attention cost and potential redundancy. Conditioning only on the finest‑scale source feature (F_{src}^K) dramatically reduces sequence length, but creates a “scale‑mismatch” problem: the model must predict coarse‑scale structure while only seeing high‑frequency details, which hampers editing quality.

To resolve this, the authors conduct an attention‑heatmap analysis on a model trained with full‑scale conditioning. They discover that the first self‑attention layer attends broadly to all source scales, establishing global layout, whereas deeper layers focus locally, refining details. This insight motivates the Scale‑Aligned Reference (SAR) module. SAR injects scale‑matched source information (the appropriate source residual for the target scale) only into the first self‑attention layer, while all subsequent layers continue to use the finest‑scale conditioning. Consequently, the model receives the necessary global context early on and refines locally with high‑frequency cues later, effectively bridging the scale gap without incurring the full computational burden.

VAREdit is fine‑tuned from a pre‑trained VAR backbone using instruction‑image pairs from the EMU‑Edit and PIE‑Bench datasets. The loss combines token classification for residual prediction and reconstruction loss for the decoded image. Experiments show that VAREdit outperforms state‑of‑the‑art diffusion editors (e.g., UltraEdit, StableDiffusion‑Edit) on both CLIP‑Score and GPT‑Score, with improvements of roughly 6–8 % on average across diverse editing scenarios (object addition, replacement, removal, material, color, style changes, and complex compositional edits).

In terms of efficiency, VAREdit edits a 512 × 512 image in 1.2 seconds on a single GPU, a 2.2× speed‑up over UltraEdit, while using roughly 30 % less memory thanks to the reduced token length and the use of KV‑cache and hybrid parallelization techniques. Ablation studies confirm that (1) full‑scale conditioning yields the best raw accuracy but is impractical; (2) finest‑scale only conditioning is fast but suffers quality loss; (3) SAR applied solely to the first self‑attention layer provides the best trade‑off; and (4) extending SAR to deeper layers offers no additional benefit.

Limitations include the current focus on 512 × 512 resolution; scaling to higher resolutions will require additional coarse‑to‑fine stages and memory‑efficient tokenization. Moreover, extremely complex textual instructions can still challenge the model’s semantic parsing. Future work aims to integrate multi‑modal feedback loops for iterative refinement, develop higher‑resolution tokenizers, and explore more sophisticated scale‑transfer mechanisms.

Overall, VAREdit demonstrates that visual autoregressive modeling, when equipped with a carefully designed scale‑aligned conditioning mechanism, can surpass diffusion‑based methods in both editing fidelity and runtime efficiency, opening a promising new direction for real‑time, high‑quality instruction‑guided image manipulation.