Canvas-of-Thought: Grounding Reasoning via Mutable Structured States

While Chain-of-Thought (CoT) prompting has significantly advanced the reasoning capabilities of Multimodal Large Language Models (MLLMs), relying solely on linear text sequences remains a bottleneck for complex tasks. We observe that even when auxiliary visual elements are interleaved, they are often treated as static snapshots within a one-dimensional, unstructured reasoning chain. We argue that such approaches treat reasoning history as an immutable stream: correcting a local error necessitates either generating verbose downstream corrections or regenerating the entire context. This forces the model to implicitly maintain and track state updates, significantly increasing token consumption and cognitive load. This limitation is particularly acute in high-dimensional domains, such as geometry and SVG design, where the textual expression of CoT lacks explicit visual guidance, further constraining the model’s reasoning precision. To bridge this gap, we introduce \textbf{Canvas-of-Thought (Canvas-CoT)}. By leveraging a HTML Canvas as an external reasoning substrate, Canvas-CoT empowers the model to perform atomic, DOM-based CRUD operations. This architecture enables in-place state revisions without disrupting the surrounding context, allowing the model to explicitly maintain the “ground truth”. Furthermore, we integrate a rendering-based critique loop that serves as a hard constraint validator, providing explicit visual feedback to resolve complex tasks that are difficult to articulate through text alone. Extensive experiments on VCode, RBench-V, and MathVista demonstrate that Canvas-CoT significantly outperforms existing baselines, establishing a new paradigm for context-efficient multimodal reasoning.

💡 Research Summary

The paper “Canvas‑of‑Thought: Grounding Reasoning via Mutable Structured States” addresses a fundamental limitation of current Chain‑of‑Thought (CoT) prompting for multimodal large language models (MLLMs). Traditional CoT treats reasoning as a linear, append‑only text stream, which forces the model to implicitly track all intermediate states within its context window. This leads to high token consumption, error‑propagation, and especially poor performance on tasks that require precise spatial reasoning such as geometry or SVG design, where textual descriptions cannot capture the full visual state.

To overcome these issues, the authors propose Canvas‑of‑Thought (Canvas‑CoT), a framework that externalizes the reasoning state into an HTML Canvas represented as a Document Object Model (DOM) tree. The LLM is re‑cast as a “Stateful Controller” that can issue atomic CRUD (Create, Read, Update, Delete) operations on DOM elements. Each operation directly modifies the external state, allowing non‑monotonic updates: a previously inserted node can be replaced or deleted without re‑generating the entire text trace. After each state transition, a headless browser renders the DOM into an image. A separate Critic model compares this rendered image with the original visual input, producing a structured feedback JSON that categorizes discrepancies into attribute errors, false existences, or spatial conflicts. This feedback acts as a “visual gradient”, guiding the next reasoning step.

Key technical components include:

1. Structured Substrate – The reasoning state S is defined as a set of vertices V (geometric points, circuit components, etc.), hierarchical edges E, and attributes P, all stored in a DOM.
2. Action Space – Four atomic actions (Insert, Replace, Modify, Delete) map a current state S_t and action a_t to a new state S_{t+1} via a deterministic transition function δ.
3. Iterative Loop – At each iteration the model receives the original instruction, the original image, the rendered current state, and the accumulated action‑feedback history. It outputs a thought trace τ_t (enclosed in ) and an action a_t (enclosed in <tool_call>).
4. Context Pruning – The textual thought trace is discarded after execution; only the persistent DOM and the latest Critic feedback are kept for the next prompt, dramatically reducing context length and eliminating noise from earlier reasoning steps.
5. Termination – The process ends when the model emits an token, indicating the external state now fully resolves the query.

The authors evaluate Canvas‑CoT on three challenging benchmarks: VCode (code‑to‑image generation), RBench‑V (visual‑language tasks), and MathVista (multimodal math problems). They compare against strong baselines including GPT‑5, Gemini 2.5/3, Claude‑4‑Opus, and multiple CoT variants (Chain‑of‑Thought, Tree‑of‑Thought, Program‑of‑Thought, Iterative Reflection). Results (Table 1) show that Canvas‑CoT consistently outperforms baselines, achieving the highest average accuracy (up to 61.2% vs. 55.4% for the best GPT‑5 CoT) and reducing token usage by roughly 15%. Qualitative analysis demonstrates that many errors are corrected with a single DOM operation rather than lengthy textual rewrites, confirming the efficiency of non‑monotonic updates. The rendering‑critique loop also successfully catches geometrically impossible configurations that pure text models would hallucinate.

Limitations are acknowledged: the current implementation is tied to HTML Canvas/SVG, so extending to full 3D engines, physics simulators, or non‑visual modalities would require additional engineering. The Critic operates at the pixel‑level, which may miss subtle numeric discrepancies. Moreover, the system relies on a deterministic parser to convert generated HTML fragments into DOM nodes; malformed output could break the pipeline, though the parser includes validation checks.

Future work suggested includes: (1) integrating more general external substrates such as graph databases or 3D scene graphs; (2) developing multimodal critics that incorporate textual, auditory, or video feedback; (3) training the LLM jointly with the CRUD action policy and the Critic to enable end‑to‑end optimization; and (4) exploring interactive UI scenarios where human users can directly edit the Canvas, enabling collaborative problem solving.

In summary, Canvas‑of‑Thought introduces a novel three‑stage loop—text generation, structured state manipulation, visual verification—that bridges the gap between symbolic reasoning and grounded visual computation. By externalizing mutable state and providing immediate visual feedback, it reduces token overhead, mitigates error propagation, and markedly improves performance on high‑dimensional multimodal tasks, establishing a new paradigm for efficient and reliable multimodal reasoning.