Investigating Text Insulation and Attention Mechanisms for Complex Visual Text Generation

In this paper, we present TextCrafter, a Complex Visual Text Generation (CVTG) framework inspired by selective visual attention in cognitive science, and introduce the “Text Insulation-and-Attention” mechanisms. To implement the selective-attention principle that selection operates on discrete objects, we propose a novel Bottleneck-aware Constrained Reinforcement Learning for Multi-text Insulation, which substantially improves text-rendering performance on the strong Qwen-Image pretrained model without introducing additional parameters. To align with the selective concentration principle in human vision, we introduce a text-oriented attention module with a novel Quotation-guided Attention Gate that further improves generation quality for each text instance. Our Reinforcement Learning based text insulation approach attains state-of-the-art results, and incorporating text-oriented attention yields additional gains on top of an already strong baseline. More importantly, we introduce CVTG-2K, a benchmark comprising 2,000 complex visual-text prompts. These prompts vary in positions, quantities, lengths, and attributes, and span diverse real-world scenarios. Extensive evaluations on CVTG-2K, CVTG-Hard, LongText-Bench, and Geneval datasets confirm the effectiveness of TextCrafter. Despite using substantially fewer resources (i.e., 4 GPUs) than industrial-scale models (e.g., Qwen-Image, GPT Image, and Seedream), TextCrafter achieves superior performance in mitigating text misgeneration, omissions, and hallucinations.

💡 Research Summary

The paper introduces TextCrafter, a novel framework for Complex Visual Text Generation (CVTG) that draws inspiration from selective visual attention in cognitive science. The authors argue that existing diffusion‑based text‑to‑image models (e.g., FLUX, SD3, Qwen‑Image) excel at rendering simple text but struggle with scenes containing multiple textual elements, often producing mis‑generated characters, omissions, or hallucinated text. To address these issues, TextCrafter implements two complementary mechanisms: Text Insulation and Text‑oriented Attention.

Text Insulation treats each textual instance as an independent object, thereby reducing cross‑instance interference. The authors formulate a Bottleneck‑aware Constrained Reinforcement Learning (BACRL) approach that uses an OCR‑based reward model during post‑training. The reward aggregates both the average similarity between generated and target texts and a bottleneck‑sensitive term that emphasizes the worst‑performing instance (the min‑operator). This design forces the model to improve the most problematic text rather than merely boosting overall averages, effectively mitigating omissions. The reward also includes a length‑based decay penalty to suppress over‑generation and hallucinations. Training is performed with a lightweight LoRA adapter (≈0.1 % of total parameters), preserving the original architecture of the strong base model (Qwen‑Image) without adding extra branches or glyph inputs.

Text‑oriented Attention introduces a Quotation‑guided Attention Gate (QAG). The gate leverages quotation marks in the prompt as spatial anchors: after smoothing the positions of quotation marks, the method retains primary peaks and applies soft binarization to create a mask that modulates the attention scores of text tokens. Consequently, each token’s attention is concentrated within the region delimited by its surrounding quotation marks, reducing cross‑text interference and aligning with the selective‑enhancement principle of human visual attention.

To evaluate the proposed mechanisms, the authors construct CVTG‑2K, a benchmark consisting of 2,000 complex prompts that feature 2–5 text regions per image, varied languages (English and Chinese), and multiple visual attributes (size, color, font). An additional harder subset (CVTG‑Hard) contains 400 prompts with even more challenging layouts. Compared with prior datasets (CreativeBench, MARIO‑Eval, AnyText‑benchmark, LongText‑Bench), CVTG‑2K offers substantially higher average word count (8.10) and character count (39.47), as well as explicit modeling of attributes and region numbers.

Experimental results show that:

1. Text Insulation alone raises OCR‑F1 scores by 4–6 % over the baseline Qwen‑Image and reduces omission rates by roughly 9 %.
2. Adding Text‑oriented Attention yields an additional 1.8 % gain in OCR‑F1 and improves human preference scores (5‑point Likert) from 3.7 to 4.3.
3. The entire pipeline runs on four A100 GPUs (40 GB each) for about 12 hours, requiring only a tiny LoRA parameter set, yet it outperforms industrial‑scale models such as GPT‑Image and Seedream that use far more compute resources.
4. On CVTG‑2K and CVTG‑Hard, TextCrafter consistently achieves lower mis‑generation, omission, and hallucination rates (30–45 % improvement) especially when five textual objects coexist.

The authors acknowledge limitations: the OCR‑based reward depends on OCR quality, which may vary across languages and fonts; the QAG relies on explicit quotation marks, so scenes without them would need an automatic anchor detector; and the reinforcement learning stage adds computational overhead despite the lightweight adapter. Future work is suggested to develop OCR‑free reward signals (e.g., CLIP similarity), learn automatic spatial anchors, and jointly optimize text and non‑text objects.

In summary, TextCrafter demonstrates that embedding cognitive‑inspired “text insulation” and “text‑oriented attention” into diffusion models can dramatically improve the fidelity of complex visual text generation while keeping model size and training cost modest. The newly released CVTG‑2K benchmark provides a rigorous testbed for future research, and the proposed methods set a new state‑of‑the‑art baseline for multi‑text rendering in generative AI.