CodeOCR: On the Effectiveness of Vision Language Models in Code Understanding

Large Language Models (LLMs) have achieved remarkable success in source code understanding, yet as software systems grow in scale, computational efficiency has become a critical bottleneck. Currently, these models rely on a text-based paradigm that treats source code as a linear sequence of tokens, which leads to a linear increase in context length and associated computational costs. The rapid advancement of Multimodal LLMs (MLLMs) introduces an opportunity to optimize efficiency by representing source code as rendered images. Unlike text, which is difficult to compress without losing semantic meaning, the image modality is inherently suitable for compression. By adjusting resolution, images can be scaled to a fraction of their original token cost while remaining recognizable to vision-capable models. To explore the feasibility of this approach, we conduct the first systematic study on the effectiveness of MLLMs for code understanding. Our experiments reveal that: (1) MLLMs can effectively understand code with substantial token reduction, achieving up to 8x compression; (2) MLLMs can effectively leverage visual cues such as syntax highlighting, improving code completion performance under 4x compression; and (3) Code-understanding tasks like clone detection exhibit exceptional resilience to visual compression, with some compression ratios even slightly outperforming raw text inputs. Our findings highlight both the potential and current limitations of MLLMs in code understanding, which points out a shift toward image-modality code representation as a pathway to more efficient inference.

💡 Research Summary

The paper “CodeOCR: On the Effectiveness of Vision Language Models in Code Understanding” investigates whether source code can be represented as rendered images and processed by multimodal large language models (MLLMs) to reduce the token cost associated with traditional text‑based code processing. The authors argue that, unlike text, images can be compressed simply by lowering resolution, allowing a substantial reduction in the number of visual tokens that modern vision‑language models such as GPT‑5 and Gemini‑3 treat at the same cost as text tokens.

To answer this question, the authors conduct a systematic empirical study across five research questions (RQs). They evaluate seven state‑of‑the‑art MLLMs (including GPT‑5‑mini2, GPT‑5‑large, Gemini‑3‑Pro, Gemini‑3‑Mini, and others) on four downstream code‑understanding tasks: code completion, code summarization, clone detection, and code question answering. For each task they vary (i) compression ratio from 1× (no compression) to 8× (12.5 % of the original token budget), (ii) rendering style (plain, syntax‑highlighted, bold), and (iii) programming language (Python and Java).

RQ1 – Visual vs. Textual Input:
Across all tasks, MLLMs ingesting code images achieve performance comparable to or better than the same models fed raw text. Notably, GPT‑5‑mini2 improves clone detection F1 by 42 % when using images, and Gemini‑3‑Pro matches or exceeds text baselines on every task. This demonstrates that code images are a viable alternative to tokenized source code.

RQ2 – Resilience to Visual Compression:
Performance degrades gracefully as resolution is lowered. Even at 8× compression (only 12.5 % of the original token count) several models still beat the raw‑text baseline. Gemini‑3‑Pro reaches 79.5 % accuracy on code QA at 8×, surpassing its 74.8 % text result. The results suggest that visual cues (color, indentation, bracket alignment) preserve enough structural information for the models to reason about code.

RQ3 – Impact of Visual Enhancements:
Syntax highlighting and bold rendering provide modest gains (1–3 % in Edit Similarity or accuracy) when compression is mild (1×–4×). At extreme compression (8×) the enhancements become invisible due to pixelation, and bold rendering can even hurt performance by further reducing character clarity. Hence visual enhancements are beneficial only within a “sweet spot” of resolution.

RQ4 – Cross‑Language Generalization:
Repeating RQ1–RQ3 on Java yields the same trends. Gemini‑3‑Pro improves Java code completion by up to 12 % Edit Similarity, and clone detection gains of 6–20 % accuracy are observed across models. This confirms that the findings are not language‑specific.

RQ5 – Information Degradation Analysis:
The authors conduct an OCR‑style reconstruction experiment, asking models to regenerate the original source from compressed images. Errors appear hierarchically: low compression introduces token‑level mistakes (missing characters or symbols), moderate compression adds line‑level errors (incorrect indentation or line breaks), and high compression leads to block‑level errors (mis‑identified function or class boundaries). Interestingly, token‑level errors often do not translate into downstream performance loss, indicating that models can infer correct semantics even when the visual signal is slightly blurred.

The paper also releases a tool called CODEOCR that renders source code into images with configurable resolution, syntax highlighting, and bolding, facilitating reproducibility and further research.

Key Contributions:

1. The first large‑scale empirical evaluation of visual code representation with MLLMs, covering multiple models, tasks, compression levels, and rendering styles.
2. Demonstration that image‑based code input can achieve comparable or superior performance to text input without any model‑specific fine‑tuning, thereby offering a token‑efficient inference pathway.
3. Introduction of CODEOCR, a practical rendering pipeline for researchers and developers.

Implications and Future Work:
While the results are promising, current MLLMs are not explicitly trained on code images, leading to variability across models and tasks. Future directions include pre‑training vision encoders on large code‑image corpora, designing adaptive rendering strategies that balance compression and visual clarity, and exploring hybrid token‑image architectures that can dynamically switch between modalities. The study opens a new avenue for reducing inference costs in software‑engineering AI applications by leveraging the compressibility of the visual modality.