Med3D-R1: Incentivizing Clinical Reasoning in 3D Medical Vision-Language Models for Abnormality Diagnosis

Developing 3D vision-language models with robust clinical reasoning remains a challenge due to the inherent complexity of volumetric medical imaging, the tendency of models to overfit superficial report patterns, and the lack of interpretability-aware reward designs. In this paper, we propose Med3D-R1, a reinforcement learning framework with a two-stage training process: Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL). During SFT stage, we introduce a residual alignment mechanism to bridge the gap between high-dimensional 3D features and textual embeddings, and an abnormality re-weighting strategy to emphasize clinically informative tokens and reduce structural bias in reports. In RL stage, we redesign the consistency reward to explicitly promote coherent, step-by-step diagnostic reasoning. We evaluate our method on medical multiple-choice visual question answering using two 3D diagnostic benchmarks, CT-RATE and RAD-ChestCT, where our model attains state-of-the-art accuracies of 41.92% on CT-RATE and 44.99% on RAD-ChestCT. These results indicate improved abnormality diagnosis and clinical reasoning and outperform prior methods on both benchmarks. Overall, our approach holds promise for enhancing real-world diagnostic workflows by enabling more reliable and transparent 3D medical vision-language systems.

💡 Research Summary

Med3D‑R1 addresses the longstanding challenge of endowing 3‑dimensional medical vision‑language models (VLMs) with genuine clinical reasoning capabilities. The authors first identify three core obstacles: (1) the intrinsic spatial and semantic complexity of volumetric CT data, (2) a positional bias in radiology reports where normal findings dominate the early sections and abnormal findings are pushed toward the end, and (3) the tendency of supervised fine‑tuning (SFT) to overfit superficial report patterns rather than learning diagnostic logic.

To overcome these issues, the paper proposes a two‑stage training pipeline. In the SFT stage, two novel modules are introduced. The Residual Alignment Mechanism (RAM) maps high‑dimensional 3D features extracted by a ViT encoder onto a set of fixed textual anchors, but does so by learning residual vectors relative to those anchors. This “anchor + residual” formulation reduces the difficulty of aligning sparse volumetric representations with dense language embeddings and improves interpretability of the intermediate space. The Abnormality Re‑Weighting (ARW) strategy quantifies the normal‑first positional bias and applies token‑level weights that amplify the contribution of abnormal tokens during loss computation, thereby counteracting the model’s propensity to default to normal predictions.

The second stage employs reinforcement learning (RL) with a newly designed Consistency Reward. Unlike prior medical RL approaches that reward only final answer correctness, this reward explicitly measures the alignment between the model’s step‑by‑step reasoning chain (e.g., slice → lesion → diagnosis) and the logical flow of the reference radiology report. The authors compute a composite score using token‑level sequence matching, cosine similarity of intermediate embeddings, and rule‑based order consistency. Policy updates are performed with Group Relative Policy Optimization (GRPO), which stabilizes learning in the high‑dimensional action space of language generation.

Experiments are conducted on two public 3D diagnostic benchmarks under the Medical Multiple‑choice Visual Question Answering (MMVQA) setting: CT‑RATE and RAD‑ChestCT. Med3D‑R1 achieves 41.92 % accuracy on CT‑RATE and 44.99 % on RAD‑ChestCT, surpassing previous state‑of‑the‑art 3D VLMs by 3–5 percentage points. Ablation studies reveal that removing RAM drops performance by 2.8 % points, omitting ARW reduces it by 1.9 % points, and excluding the Consistency Reward leads to comparable final accuracy but a substantial decline in reasoning coherence scores (over 30 % lower). Visualizations show that RAM focuses attention on clinically relevant anatomical regions, while ARW mitigates the normal‑first bias in token predictions.

The paper acknowledges limitations: the slice‑wise processing incurs significant memory and compute overhead, and the textual anchors are currently handcrafted for CT‑specific terminology, limiting generality across modalities. Future work is suggested to explore multi‑scale 3D feature aggregation, ontology‑driven universal anchors, and human‑in‑the‑loop evaluations with radiologists to validate clinical utility.

In summary, Med3D‑R1 introduces a principled residual alignment, bias‑aware re‑weighting, and reasoning‑consistent reinforcement learning framework that together advance 3D medical vision‑language models from pattern memorization toward transparent, clinically grounded diagnostic reasoning, paving the way for more trustworthy AI assistance in radiology workflows.