GlyphBanana: Advancing Precise Text Rendering Through Agentic Workflows

Despite recent advances in generative models driving significant progress in text rendering, accurately generating complex text and mathematical formulas remains a formidable challenge. This difficulty primarily stems from the limited instruction-following capabilities of current models when encountering out-of-distribution prompts. To address this, we introduce GlyphBanana, alongside a corresponding benchmark specifically designed for rendering complex characters and formulas. GlyphBanana employs an agentic workflow that integrates auxiliary tools to inject glyph templates into both the latent space and attention maps, facilitating the iterative refinement of generated images. Notably, our training-free approach can be seamlessly applied to various Text-to-Image (T2I) models, achieving superior precision compared to existing baselines. Extensive experiments demonstrate the effectiveness of our proposed workflow. Associated code is publicly available at https://github.com/yuriYanZeXuan/GlyphBanana.

💡 Research Summary

GlyphBanana addresses the persistent problem that diffusion‑based text‑to‑image (T2I) models struggle with out‑of‑distribution (OOD) prompts, especially when rendering rare characters, complex Chinese glyphs, or intricate scientific formulas. Existing solutions fall into two camps: training‑based methods (e.g., LoRA fine‑tuning, encoder fine‑tuning) that require large annotated datasets and suffer from limited generalization, and training‑free methods that impose a strong glyph prior, often degrading the overall visual style.

The proposed system introduces an agentic workflow that leverages plug‑and‑play auxiliary tools—vision‑language models, layout planners, font controllers, and formula renderers—to inject precise glyph information into both the latent space and the attention maps of any diffusion model without any additional training. The pipeline consists of four tightly coupled stages:

1. Extraction – A VLM parses the user prompt into target text content T and a style description S. This decomposition supplies the ground‑truth needed for downstream modules.

2. Draft Preview – A base T2I model generates an initial image matching the desired style. A layout planner, equipped with text‑grounding utilities, produces a detailed typography plan P that specifies font family, weight, size, color, bounding‑box coordinates, and rotation angles for each glyph.

3. Glyph Injection – This core stage combines two mechanisms:
   Frequency Decomposition: The latent representation zₜ is split into low‑frequency (LF) and high‑frequency (HF) components via Gaussian blur. An Otsu‑derived mask M isolates glyph‑covered tokens, allowing the high‑frequency glyph template z_tpl to replace the corresponding HF part while preserving LF background information. An injection window