Synthetic Vasculature and Pathology Enhance Vision-Language Model Reasoning

Vision-Language Models (VLMs) offer a promising path toward interpretable medical diagnosis by allowing users to ask about clinical explanations alongside predictions and across different modalities. However, training VLMs for detailed reasoning requires large-scale image-text datasets. In many specialized domains, for example in reading Optical Coherence Tomography Angiography (OCTA) images, such precise text with grounded description of pathologies is scarce or even non-existent. To overcome this bottleneck, we introduce Synthetic Vasculature Reasoning (SVR), a framework that controllably synthesizes images and corresponding text, specifically: realistic retinal vasculature with Diabetic Retinopathy (DR) features: capillary dropout, microaneurysms, neovascularization, and tortuosity, while automatically generating granular reasoning texts. Based on this we curate OCTA-100K-SVR, an OCTA image-reasoning dataset with 100,000 pairs. Our experiments show that a general-purpose VLM (Qwen3-VL-8b) trained on the dataset achieves a zero-shot balanced classification accuracy of 89.67% on real OCTA images, outperforming supervised baselines. Through human expert evaluation we also demonstrate that it significantly enhances explanation quality and pathology localization on clinical data.

💡 Research Summary

This paper tackles the critical shortage of large, finely annotated medical image‑text datasets needed to train vision‑language models (VLMs) for detailed reasoning, focusing on Optical Coherence Tomography Angiography (OCTA) for diabetic retinopathy (DR). The authors introduce Synthetic Vasculature Reasoning (SVR), a two‑stage pipeline that first generates realistic retinal vasculature graphs with controllable DR pathologies—capillary dropout, microaneurysms, neovascularization, and increased tortuosity—and then converts these graphs into high‑fidelity OCTA images using a pretrained GAN. For each synthetic image, structured pathology metadata is turned into a deterministic template paragraph describing the presence, severity, and spatial relationship of each lesion. This template is fed to a teacher large language model (GPT‑5) which rewrites it into a diverse, chain‑of‑thought (CoT) explanation while preserving all clinical facts. The result is a massive paired dataset of 100,000 synthetic OCTA images and corresponding reasoning texts, named OCTA‑100K‑SVR.

Training proceeds in two phases. In the first phase, the general‑purpose Qwen3‑VL‑8b‑Instruct model is fine‑tuned on the synthetic pairs, updating only the vision encoder and multimodal projection layers while freezing the language backbone. This allows the model to learn OCTA‑specific visual features and their alignment to language without losing its broad linguistic capabilities. In the second phase, the pretrained checkpoint is further fine‑tuned on real OCTA data (the public OCTA‑500 set and an in‑house collection of 1,286 scans) using actual DR labels and expert‑written explanations. This end‑to‑end fine‑tuning adapts decision boundaries and linguistic style to the clinical domain while retaining the pathology‑aware visual grounding acquired from synthetic data.

The authors evaluate the approach on three fronts: (1) zero‑shot DR vs. healthy classification on OCTA‑500, (2) three‑class staging (healthy, non‑proliferative DR, proliferative DR) on the in‑house dataset, and (3) explanation quality. Baselines include traditional CNN/GNN classifiers (ResNet‑18, vessel‑graph GNN), several off‑the‑shelf VLMs (vanilla Qwen3‑VL‑8b/30b, LLaMA‑3.2‑11B‑VL, LLaVA‑NEXT‑8B), a graph‑knowledge‑enhanced VLM (Qwen3‑VL‑8b‑GFT), and a VLM fine‑tuned only on class labels. Classification performance is measured by balanced accuracy (mean per‑class recall). Explanation quality is assessed automatically by GPT‑5 (scoring helpfulness, clinical accuracy, localization accuracy, relevance) and manually by two ophthalmologists who rank explanations on the same criteria.

Results show that the SVR‑pretrained model achieves a zero‑shot balanced accuracy of 89.67 % on real OCTA images, surpassing all baselines. After fine‑tuning on real data (SVR‑FT), the model maintains top performance, especially improving proliferative DR detection by over 7 percentage points compared to other VLMs. Explanation scores are consistently higher for SVR‑based models; both GPT‑5 and human judges rate their pathology localization and clinical relevance markedly better. A scaling study demonstrates a strong positive correlation between the size of the synthetic training set (1 k to 100 k) and both classification accuracy and GPT‑5 explanation scores, confirming that larger synthetic corpora yield more capable VLMs.

The paper’s contributions are threefold: (1) a controllable 3‑D vascular graph simulator that can embed four clinically relevant DR lesions with interpretable hyper‑parameters, (2) an automated pipeline that transforms simulator metadata into diverse, medically accurate CoT text using a powerful LLM, and (3) a two‑stage training strategy that leverages massive synthetic data to bootstrap VLM reasoning before domain‑specific fine‑tuning. The authors discuss limitations, noting that while visual realism is high, subtle flow‑signal nuances of real OCTA are not fully captured, and that reliance on GPT‑5 for text generation could be replaced by medically specialized LLMs in the future. They also suggest that the SVR framework can be extended to other ophthalmic conditions (e.g., age‑related macular degeneration, glaucoma) and more broadly to other imaging modalities such as histopathology or radiology.

In summary, Synthetic Vasculature Reasoning demonstrates that high‑quality, pathology‑aware synthetic image‑text pairs can effectively pre‑train VLMs for medical diagnosis, dramatically reducing the need for costly expert annotations while delivering state‑of‑the‑art classification and explanation performance on real OCTA data. This work paves the way for scalable, interpretable AI systems across a wide range of specialized medical imaging domains.