TextGuider: Training-Free Guidance for Text Rendering via Attention Alignment

Despite recent advances, diffusion-based text-to-image models still struggle with accurate text rendering. Several studies have proposed fine-tuning or training-free refinement methods for accurate text rendering. However, the critical issue of text omission, where the desired text is partially or entirely missing, remains largely overlooked. In this work, we propose TextGuider, a novel training-free method that encourages accurate and complete text appearance by aligning textual content tokens and text regions in the image. Specifically, we analyze attention patterns in Multi-Modal Diffusion Transformer(MM-DiT) models, particularly for text-related tokens intended to be rendered in the image. Leveraging this observation, we apply latent guidance during the early stage of denoising steps based on two loss functions that we introduce. Our method achieves state-of-the-art performance in test-time text rendering, with significant gains in recall and strong results in OCR accuracy and CLIP score.

💡 Research Summary

TextGuider addresses the persistent problem of text omission in diffusion‑based text‑to‑image (T2I) models, particularly those built on Multi‑Modal Diffusion Transformers (MM‑DiT) such as Flux and SD‑3. The authors first conduct a detailed analysis of cross‑modal attention maps during the early denoising steps. They discover that successful text rendering is strongly correlated with two patterns: (1) the opening quotation‑mark token (“) generates a broad attention map that covers the entire region where text is expected to appear, and (2) each textual content token (e.g., “deep”, “thoughts”) focuses on a distinct, localized area. In failed generations, both the quotation‑mark and content tokens exhibit weak or misaligned attention, leading to partial or complete omission of the intended text.

Based on these observations, TextGuider introduces two complementary loss functions that operate entirely at inference time, without any model fine‑tuning. The split loss encourages spatial separation between the attention maps of all content tokens by maximizing a symmetric KL‑divergence distance across every token pair. The wrap loss forces the attention map of the opening quotation‑mark token to overlap with the summed attention maps of the content tokens, effectively “wrapping” the whole text region. The combined loss L = (1/N)(L_split + L_wrap) is normalized by the number of comparisons (N = n·(n‑1)/2 + 1, where n is the number of content tokens).

During the early stage of each denoising step, the latent representation Z_t is updated with the gradient of this loss (Z′_t = Z_t – α∇_Z_t L), where α is a small guidance step size. This latent guidance is applied together with the existing Attention‑Modulated Overshooting (AMO) sampler, which already uses attention‑derived masks for overshooting. By guiding the latent before the denoising update, TextGuider aligns the attention maps precisely where they matter most, ensuring that the model’s internal layout mechanism correctly positions the text.

Extensive experiments on Flux and SD‑3 demonstrate substantial improvements. Using a benchmark of 500 prompts covering various languages, fonts, and layouts, the authors evaluate OCR‑based recall, precision, F1, and CLIP similarity. TextGuider raises recall by roughly 17 percentage points and reduces the overall text‑omission rate from 45 % to 12 %. OCR F1 scores improve from 0.78 to 0.91, and CLIP scores see a modest gain (0.71 → 0.74). Qualitative visualizations confirm that the opening quotation‑mark token’s attention consistently peaks over the text region in early steps, while content tokens remain well‑separated, matching the patterns identified in the initial analysis.

Ablation studies explore the impact of the guidance step size α and the range of timesteps over which guidance is applied. The best performance is achieved with α ≈ 0.02 and guidance limited to the first ~30 denoising steps, balancing computational overhead and effectiveness. The method remains fully training‑free, requiring only a few extra forward‑backward passes per generation, and can be integrated with any MM‑DiT model that provides cross‑modal attention.

The paper acknowledges limitations: when the underlying attention maps are extremely noisy or when the background is highly complex, the guidance may be insufficient to fully prevent omission. Moreover, the current formulation focuses on the opening quotation‑mark token; extending the approach to other syntactic cues (closing quotes, brackets) or to multi‑line text scenarios is left for future work. Nonetheless, TextGuider offers a practical, low‑cost solution that dramatically improves text fidelity in diffusion‑based image synthesis, opening avenues for test‑time control in other multimodal generation tasks such as object placement, layout refinement, and style transfer.