The Cost of Reasoning: Chain-of-Thought Induces Overconfidence in Vision-Language Models

The Cost of Reasoning: Chain-of-Though t Induces Ov erconfidence in Vision-Language Mo dels Rob ert W elc h 1 , 2 , Emir Kon uk 1 , 2 , and Kevin Smith 1 , 2 1 KTH Ro yal Institute of T echnology , Sto c kholm, Sweden 2 Science for Life Lab oratory , Sto c kholm, Sw eden rwe2@kth.se Abstract. Vision-language mo dels (VLMs) are increasingly deplo yed in high-stak es settings where reliable uncertain ty quan tification (UQ) is as imp ortan t as predictive accuracy . Extended reasoning via chain- of-though t (CoT) prompting or reasoning-trained models has become ubiquitous in mo dern VLM pip elines, y et its effect on UQ reliability re- mains po orly understoo d. W e show that reasoning consistently degrades the qualit y of most uncertain ty estimates, even when it impro ves task accuracy . W e identify implicit answ er conditioning as the primary mech- anism: as reasoning traces con verge on a conclusion b efore the final an- sw er is generated, tok en probabilities increasingly reflect consistency with the mo del’s own reasoning trace rather than uncertaint y ab out correct- ness. In effect, the mo del b ecomes ov erconfident in its answer. In con- trast, agreement-based consistency remains robust and often improv es under reasoning, making it a practical c hoice for uncertaint y estimation in reasoning-enabled VLMs. Keyw ords: Vision-Language Mo dels · Chain-of-Though t · Uncertaint y Quan tification 1 In tro duction Vision-language models (VLMs) are increasingly deplo yed in high-stak es set- tings where reliable predictions are as imp ortant as accurate ones, from in- terpreting medical images [15] to autonomous na vigation [29]. Modern VLM pip elines increasingly rely on c hain-of-thought (CoT) prompting and reasoning- orien ted models that generate in termediate, step-b y-step inference b efore pro- ducing a final answ er. Reasoning substantially improv es p erformance on chal- lenging m ultimo dal benchmarks requiring multi-step visual and mathematical reasoning [17, 32]. Ho wev er, accuracy alone is insufficient for safe deploymen t. Sele ctive gener- ation addresses this issue by ensuring that the mo dels can abstain when un- certain [2]. In this setting, uncertaint y quality is determined by ho w well con- fidence scores rank correct predictions abov e incorrect ones, and is typically ev aluated using ranking-based metrics suc h as the Prediction Rejection Ratio (PRR) [19, 20] and AUGR C [25]. 2 R. W elc h et al. User Prompt Q: Name the t yp e of plant. Ground T ruth vine VLM Responses No Reasoning vertical garden Reasoning 1 The image shows a vertical garden with lush green foliage , including broad leaves and some reddish variegation . vertical garden Reasoning 2 The image shows a vertical garden with lush green foliage . This appearance is characteristic of a vertical garden , commonly using species like pothos , philodendron , or ivy . vertical garden Answer token likelihood Figure 1: Example: image (left), highligh ted tok en traces (righ t), and a color scale. 1 Fig. 1: Example illustrating reasoning-induced ov erconfidence. Color intensit y reflects answer-tok en likelihoo d, which increases as the reasoning trace conv erges. As the model generates a c hain-of-thought explanation, the evolving reasoning increasingly constrains the answer tokens (“v ertical garden”), making predictions ov erly confiden t, ev en when the prediction is incorrect (ground truth: vine ). W e refer to this effect as implicit answer c onditioning : the model progressively shifts from groun ding the answer in the image to reinforcing its own reasoning trace, which in turn degrades reliability of many uncertaint y estimates. Giv en the widespread adoption of reasoning, a natural question arises. How do es reasoning affect uncertain ty estimation in m ulti-mo dal generation? One migh t expect that deliberate reasoning impro ves b oth accuracy and UQ relia- bilit y . How ever, across the VLM families and b enc hmarks we ev aluated, w e find that this intuition fails for most uncertaint y estimates. While reasoning typically impro ves task accuracy , it degrades the ranking quality of uncertaint y measures that rely on answ er-token lik eliho o ds, including maxim um sequence probabil- it y , p erplexity , mean token entrop y , and Mon te Carlo sequence en tropy [4, 14]. The issue is not merely that additional conditioning c hanges token probabilities (whic h is exp ected) but that reasoning changes what the answer-tok en likelihoo d represen ts. Rather than reflecting uncertaint y ab out correctness, it increasingly captures in ternal semantic commitmen t to the mo del’s own reasoning trace. In m ultimo dal settings, this misalignment is esp ecially consequen tial: VLMs must ground their conclusions in perceptual evidence that may remain am biguous ev en as the reasoning trace conv erges. T oken-lev el probabilities can signal con- sistency with the mo del’s internal reasoning while failing to capture uncertaint y ab out the visual signal. As a result, mo dels may abstain less often on difficult or visually am biguous samples, pro ducing ov erconfiden t errors despite improv ed a verage accuracy . Our analysis suggests that this arises from what we term implicit answer c on- ditioning : as a reasoning trace unfolds, it progressiv ely constrains the conditional The Cost of Reasoning in VLMs 3 distribution ov er the forthcoming answer tokens. By the time the final answer is generated, the mo del has already committed to a sp ecific conclusion within its o wn reasoning trace, making the answer highly predictable. In la yman’s terms, it talks itself in to its answer. This inflates answer-tok en likelihoo ds, whether or not the conclusion is correct. Imp ortan tly , this effect is not attributable to verbosity: longer reasoning traces are, on av erage, asso ciated with low er confidence. Rather, it is the conditioning effect of the reasoning con text (i.e. the answ er implicit in the reasoning) that concentrates probability mass on the predicted answer. W e pro vide con trolled masking interv entions and reasoning trace analyses that offer evidence consistent with this explanation. Imp ortan tly , not all uncertaint y estimates are equally affected. Those that do not rely directly on answer-tok en likelihoo ds, such as agreement-based con- sistency [27], whic h measures agreement across sampled answ ers via a ma jority v ote, remain stable and often improv e under reasoning. Asking the mo del to self- rep ort confidence [10, 24] shows mixed b eha vior. This pap er mak es the following con tributions: 1. W e empirically demonstrate that reasoning degrades the qualit y of uncer- tain ty estimates that rely on answer-tok en likelihoo d (A TL). 2. W e provide evidence p oin ting to wards implicit answer c onditioning as a key driv er of this degradation. 3. W e show that applying agreement-based consistency is robust under reason- ing, offering a practical solution for UQ in reasoning VLMs. 2 Related W ork The interaction b etw een chain-of-though t reasoning and uncertaint y estimation has recently b een explored in text-only large language m odels. Y o on et al. [31] rep ort that CoT-induced “slow-thinking” behaviours can improv e self-rep orted confidence calibration, while Lyu et al. [18] study whether consistency enhances calibration in LLMs. Lanham et al. [16] inspect reasoning traces to analyze their causal role in shaping mo del predictions. Ho w ever, these w orks fo cus on calibration or reasoning faithfulness in text-only settings. They do not examine ho w reasoning alters the pragmatic ranking reliability of uncertaint y estimates or the selective generation setting. Crucially , prior studies do not consider multimodal mo dels, where reasoning m ust reconcile visual p erceptual uncertain t y with linguistic inference. In mul- timo dal mo dels, the reasoning pro cess can con verge on a confiden t conclusion ev en if the visual evidence is unclear, pro ducing mismatc hes b etw een in ternal confidence and p erceptual uncertain ty that do not o ccur in text-only mo dels. T o our knowledge, no prior work analyzes how reasoning alters uncertain ty ranking in multimodal generation. Chain-of-though t prompting [28] and reasoning-nativ e (“Thinking”) mod- els [7, 8] generate in termediate reasoning traces prior to pro ducing final answ ers, 4 R. W elc h et al. and are no w widely adopted. Self-Consistency [27] aggregates m ultiple reason- ing paths via ma jority voting. Despite their widespread use, their impact on uncertain ty estimation remains largely unexplored. Uncertain ty Estimation in Language and Vision-Language Mo dels. Comprehensiv e b enc hmarks such as LM-Polygraph [4] and subsequent large-scale ev aluations [26, 30] compare a wide range of white-b o x and black-box uncertaint y metho ds for LLMs. These studies, ho wev er, ev aluate static uncertaint y quantifi- cation for non-reasoning settings and do not analyze ho w reasoning b ehaviour systematically mo difies them. In multimodal settings, uncertaint y for VLMs has thus far b een studied via conformal prediction on multiple-c hoice tasks without reasoning [13]. While use- ful, conformal metho ds assume fixed answer sets and do not address op en-ended generation or tok en-lev el lik eliho o d-based uncertaint y . In this paper, w e ana- lyze a broad sp ectrum of uncertain ty measures including token-lik eliho o d-based, agreemen t-based, and introspective measures for multimodal settings. Bey ond b enc hmark comparisons, prior w ork explores b ehavioural pro xies for uncertaint y . Khan and F u [11] examine answer consistency under prompt rephrasing for selective visual question answ ering. Devic et al. [3] prop ose rea- soning trace length as an uncertain ty signal in reasoning-oriented text-only mo d- els. While related, these approaches do not analyze ho w reasoning changes the seman tics of token-lev el likelihoo d or its reliability as an uncertaint y estimate in m ultimo dal mo dels. Kuhn et al. [14] in tro duce Seman tic Entrop y , whic h esti- mates predictive uncertain ty by sampling multiple generations, clustering them in to seman tic equiv alence classes using an external natural language inference (NLI) mo del, and computing entrop y ov er clusters. Seman tic Entrop y captures seman tic diversit y across sampled outputs, whereas our analysis fo cuses on how reasoning distorts token-lev el likelihoo d within a single conditioned generation. Moreo ver, Seman tic En tropy relies on an auxiliary NLI oracle; we instead ev alu- ate uncertaint y signals computed directly from mo del outputs to enable consis- ten t m ultimo dal comparison. Kada v ath et al. [10] demonstrate that mo dels can estimate the probabilit y that a giv en answ er is correct via the P(T rue) self-probing signal. Our Self- Rep orted Confidence measure is closely related; follo wing [24], w e normalize this score and ev aluate it under ranking-based selective generation metrics. Ev aluating Uncertain ty Under Selective Generation. W e ev aluate un- certain ty within the selectiv e generation framework, where a model ma y ab- stain when its confidence falls below a threshold. This paradigm reflects safet y- critical deplo yment and assesses unc ertainty by how wel l it r anks c orr e ct pr e- dictions ab ove inc orr e ct ones under abstention . Metrics suc h as the Area Un- der the Risk–Cov erage Curve (AUR C) [5] and the Prediction Rejection Ratio (PRR) [4, 20] directly quantify this ranking qualit y . T raditional calibration met- rics such as Expected Calibration Error (ECE) are less informative here since man y effective uncertain ty signals are not calibrated probabilities [23]. This framew ork therefore provides a pragmatic and mo del-agnostic ev aluation of un- The Cost of Reasoning in VLMs 5 certain ty quality . W e build on it to show that reasoning systematically alters the reliabilit y of token-lev el uncertaint y signals. 3 Metho d Selectiv e Generation F ramew ork W e ev aluate uncertaint y under reasoning within the selective generation framework, where a mo del may abstain from an- sw ering when its confidence falls below a threshold. In this setting, the mo del pro duces answ ers for only a subset of inputs (co verage), rejecting uncertain pre- dictions and trading off cov erage against error rate (risk). Giv en a confidence score g ( x ) for input x , predictions are accepted only when g ( x ) ≥ τ for some threshold τ , otherwise w e abstain from providing an answ er. V arying τ rev eals a trade-off b etw een cov erage level and risk, and provides a pragmatic ev aluation of how effectiv ely an uncertain ty estimate identifies unreliable predictions. 3.1 Uncertain ty Estimation Metho ds Deco der-st yle VLMs output a probability distribution ov er a vocabulary V con- taining h undreds of thousands of tokens. T o decide whether to answer or ab- stain, this distribution must b e reduced to a scalar uncertaint y score. W e ev al- uate established uncertain ty estimation strategies, grouped in to three categories: Answ er-T oken Likelihoo d (A TL) Estimates, Self-Probing Estimates, and Agreement- Based Estimates. Answ er-T ok en Likelihoo d (A TL) Estimates. These measures compress tok en-level probabilities into a single scalar calculated ov er the answ er tokens. Let y = ( y 1 , . . . , y L ans ) denote the answ er sequence, where y l is the l -th token. Maximum Se quenc e Pr ob ability (MSP). The joint probability of the generated answ er sequence is used directly: P ( y | x, θ ) = L ans Y l =1 P ( y l | y Question: {question} Options: {options} Without CoT answer mo de only Respond ONLY in the following wrapped format: With CoT only First, think step by step and briefly describe the cues you would use to select the correct option, then answer. Respond ONLY in the following wrapped format: your thought process F ormat rules (b oth) Replace with the chosen option text, copied verbatim from the list of options. Include both opening and closing tags exactly once. IMPORTANT: Any deviation from this exact tag format is incorrect. The Cost of Reasoning in VLMs 19 Self-Rep orted Certaint y Prompt (Without vs. With CoT) Shared con text (b oth) You are a VQA assistant. You previously answered the question: {question} with the answer: {answer} out of the possible options: {mc_options}. Your task is now to self-report your certainty about your answer. Without CoT answer mo de only Are you confident in your answer? Respond ONLY in the following wrapped format: yes or no With CoT only First, think step by step and briefly explain why you are or are not confident in your answer. Then, answer: Are you confident in your answer? Respond ONLY in the following wrapped format: your reasoning yes or no F ormat rules (b oth) Do not add any text outside the tags. Do not add spaces or punctuation inside the yes/no tags. 20 R. W elch et al. Dataset Instructions OK-V QA When giving the final answer, give a short, direct answer using 1-2 words. Use the most common, canonical name. Examples: Q: What sort of vehicles uses this item? A: firetruck Q: How many chromosomes do these creatures have? A: 23 Q: What material is this vase made out of? A: ceramic Q: What is the man doing? A: standing Math Vista If options are listed in the question, choose one option and return exactly that option text in the final answer. Do not include equations, units, or explanation in the final answer. B Deco ding parameters F or the all mo dels, we uniformly apply the deco ding parameters detailed in T able T ab. 4 across all uncertaint y measures and exp eriments. Deco ding parameters for Qwen3-VL mo dels are set following the recommended settings rep orted in the Qw en3-VL technical rep ort [1]. F or Gemma-3, the technical rep ort do es not prescrib e decoding parameters; w e therefore adopt the default recommended settings used in Google’s public developer in terfaces and do cumentation. W e set the maximum generated tok ens sufficiently large (20k) to ensure that no generated resp onses are truncated for any experiments. The Cost of Reasoning in VLMs 21 T able 4: Decoding hyperparameters used for eac h mo del family . Mo del T emperature T op- p T op- k Presence P enalty Gemma3 Series 1.0 0.95 64 1.0 Qw en3-VL-8B-Instruct 1.0 1.0 40 2.0 Qw en3-VL-8B-Thinking 1.0 0.95 20 1.5 Qw en3-VL-32B-Instruct 0.7 0.80 20 1.5 C K-Ablation W e conduct an ablation study on Math Vista across different v alues of k (the n umber of sampled answers or reasoning traces) using CoT prompting with Qw en3-VL-8B-Instruct. Increasing k consistently impro ves agreement-based con- sistency , whic h b enefits from additional sampled reasoning paths. In con trast, the effect of increasing k on MC-SE and MC-NSE is less consisten t: while mo dest im- pro vemen ts are observ ed at larger k, their p erformance v aries across intermediate v alues. Overall, these results suggest that agreement-based consistency more reli- ably b enefits from additional sampling than token-lik eliho od–based Monte Carlo estimates. W e note that b etw een runs, the predictive performance c hanges, as the final answer is determined b y ma jority vote. (a) AUGR C across k v alues. (b) PRR across k v alues. Fig. 4: Effect of sample size k on uncertaint y quality . Lo wer A UGRC is better, higher PRR is b etter. 22 R. W elch et al. D Selectiv e Generation Results on all datasets T able 5: Selectiv e Generation Results on Oxford Pets No CoT CoT ∆ (CoT − No CoT) Type UQ Estimate AUGR C ↓ PRR ↑ Spear. ↑ AUGR C ↓ PRR ↑ Spear. ↑ AUGR C PRR Spear. Gemma3-4B-IT Single-Gener ation (Acc: 65.2 / 70.4) MSP 0.122 0.498 0.380 0.139 0.200 0.068 0.018 -0.297 -0.312 PPL 0.121 0.498 0.384 0.141 0.186 0.054 0.020 -0.312 -0.330 MTE 0.118 0.521 0.404 0.136 0.219 0.090 0.018 -0.302 -0.314 SRC 0.121 0.511 0.388 0.104 0.504 0.333 -0.016 -0.007 -0.055 Multi-Gener ation (Acc: 65.4 / 71.8) MC-SE 0.122 0.497 0.375 0.123 0.296 0.136 0.002 -0.201 -0.240 MC-NSE 0.121 0.498 0.380 0.125 0.288 0.123 0.004 -0.210 -0.257 Consistency 0.147 0.475 0.295 0.112 0.283 0.313 -0.034 -0.192 0.018 Qwen3-VL-32B-IT Single-Gener ation (Acc: 82.2 / 86.4) MSP 0.046 0.644 0.386 0.076 0.010 -0.082 0.030 -0.635 -0.467 PPL 0.049 0.534 0.395 0.076 0.012 -0.082 0.027 -0.522 -0.477 MTE 0.046 0.683 0.392 0.076 0.016 -0.080 0.030 -0.668 -0.472 SRC 0.039 0.770 0.457 0.048 0.319 0.289 0.009 -0.451 -0.168 Multi-Gener ation (Acc: 81.8 / 86.4) MC-SE 0.047 0.659 0.399 0.076 -0.002 -0.077 0.029 -0.661 -0.476 MC-NSE 0.047 0.659 0.399 0.076 -0.001 -0.077 0.029 -0.660 -0.476 Consistency 0.081 0.382 0.222 0.065 0.328 0.346 -0.016 -0.054 0.124 T able 6: Selectiv e Generation Results on OK-VQA No CoT CoT ∆ (CoT − No CoT) Type UQ Estimate AUGR C ↓ PRR ↑ Spear. ↑ AUGR C ↓ PRR ↑ Spear. ↑ AUGR C PRR Spear. Gemma3-4B-IT Single-Gener ation (Acc: 38.8 / 35.4) MSP 0.270 0.333 0.266 0.294 0.244 0.218 0.024 -0.090 -0.048 PPL 0.278 0.291 0.207 0.305 0.188 0.135 0.027 -0.103 -0.072 MTE 0.276 0.297 0.222 0.303 0.192 0.152 0.027 -0.105 -0.071 SRC 0.270 0.328 0.268 0.289 0.304 0.261 0.018 -0.025 -0.007 Multi-Gener ation (Acc: 39.0 / 37.0) MC-SE 0.270 0.336 0.262 0.285 0.279 0.226 0.015 -0.057 -0.036 MC-NSE 0.279 0.290 0.198 0.298 0.213 0.130 0.019 -0.077 -0.068 Consistency 0.273 0.239 0.274 0.264 0.376 0.392 -0.008 0.137 0.118 Qwen3-VL-32B-IT Single-Gener ation (Acc: 57.9 / 51.8) MSP 0.156 0.536 0.417 0.196 0.394 0.335 0.040 -0.141 -0.083 PPL 0.158 0.519 0.402 0.199 0.359 0.309 0.041 -0.160 -0.093 MTE 0.156 0.526 0.413 0.199 0.358 0.308 0.043 -0.168 -0.105 SRC 0.168 0.358 0.341 0.195 0.391 0.352 0.028 0.033 0.011 Multi-Gener ation (Acc: 58.1 / 52.9) MC-SE 0.156 0.530 0.407 0.186 0.463 0.368 0.030 -0.067 -0.039 MC-NSE 0.158 0.514 0.392 0.189 0.434 0.344 0.031 -0.080 -0.048 Consistency 0.182 0.225 0.264 0.190 0.346 0.372 0.007 0.121 0.108 The Cost of Reasoning in VLMs 23 T able 7: Selectiv e Generation Results on MMMU-Pro-Vision No CoT CoT ∆ (CoT − No CoT) Type UQ Estimate AUGR C ↓ PRR ↑ Spear. ↑ AUGR C ↓ PRR ↑ Spear. ↑ AUGR C PRR Spear. Gemma3-4B-IT Single-Gener ation (Acc: 19.0 / 18.9) MSP 0.401 0.118 0.037 0.398 0.036 0.069 -0.003 -0.082 0.032 PPL 0.401 0.116 0.035 0.398 0.036 0.069 -0.003 -0.081 0.034 MTE 0.398 0.129 0.059 0.397 0.039 0.075 -0.001 -0.090 0.016 SRC 0.400 0.077 0.047 0.394 0.122 0.108 -0.006 0.045 0.060 Multi-Gener ation (Acc: 18.6 / 20.3) MC-SE 0.401 0.116 0.056 0.390 0.072 0.076 -0.011 -0.044 0.019 MC-NSE 0.401 0.115 0.056 0.390 0.071 0.075 -0.011 -0.044 0.019 Consistency 0.406 0.016 0.042 0.385 0.088 0.123 -0.021 0.072 0.081 Qwen3-VL-32B-IT Single-Gener ation (Acc: 42.9 / 56.7) MSP 0.227 0.521 0.415 0.198 0.124 0.210 -0.028 -0.397 -0.205 PPL 0.227 0.521 0.415 0.198 0.124 0.210 -0.028 -0.397 -0.205 MTE 0.225 0.544 0.424 0.181 0.303 0.245 -0.044 -0.242 -0.179 SRC 0.238 0.459 0.336 0.189 0.305 0.193 -0.049 -0.154 -0.143 Multi-Gener ation (Acc: 43.8 / 58.6) MC-SE 0.226 0.481 0.387 0.180 0.201 0.229 -0.046 -0.280 -0.158 MC-NSE 0.226 0.481 0.387 0.179 0.201 0.229 -0.046 -0.279 -0.158 Consistency 0.263 0.059 0.214 0.168 0.295 0.322 -0.094 0.236 0.107 T able 8: Selectiv e Generation Results on Math Vista Mini T est Split No CoT CoT ∆ (CoT − No CoT) Type UQ Estimate AUGR C ↓ PRR ↑ Spear. ↑ AUGR C ↓ PRR ↑ Spear. ↑ AUGR C PRR Spear. Gemma3-4B-IT Single-Gener ation (Acc: 34.7 / 41.9) MSP 0.292 0.317 0.256 0.269 0.202 0.150 -0.022 -0.115 -0.106 PPL 0.300 0.239 0.192 0.275 0.152 0.110 -0.025 -0.087 -0.082 MTE 0.302 0.229 0.182 0.276 0.152 0.107 -0.026 -0.077 -0.074 SRC 0.340 0.022 -0.094 0.289 0.083 0.012 -0.051 0.060 0.106 Multi-Gener ation (Acc: 35.2 / 43.9) MC-SE 0.289 0.318 0.252 0.264 0.217 0.120 -0.026 -0.100 -0.132 MC-NSE 0.300 0.238 0.178 0.268 0.197 0.092 -0.032 -0.041 -0.086 Consistency 0.296 0.237 0.249 0.230 0.441 0.359 -0.066 0.203 0.111 Qwen3-VL-32B-IT Single-Gener ation (Acc: 70.9 / 80.8) MSP 0.095 0.627 0.388 0.071 0.338 0.218 -0.023 -0.289 -0.170 PPL 0.100 0.586 0.347 0.075 0.289 0.190 -0.025 -0.297 -0.158 MTE 0.100 0.585 0.349 0.075 0.292 0.184 -0.024 -0.293 -0.165 SRC 0.096 0.491 0.377 0.048 0.711 0.425 -0.048 0.220 0.047 Multi-Gener ation (Acc: 70.7 / 82.9) MC-SE 0.096 0.621 0.383 0.065 0.445 0.194 -0.032 -0.176 -0.189 MC-NSE 0.103 0.572 0.329 0.067 0.403 0.168 -0.036 -0.169 -0.161 Consistency 0.126 0.287 0.285 0.046 0.501 0.480 -0.080 0.214 0.195 24 R. W elch et al. E A dditional Figures for UQ Shifts (a) OK-VQA (b) MMMU-Pro-Vision (c) Oxford Pets Fig. 5: Sample-level confidence shifts under Chain-of-Thought prompting for Qw en3- VL-32B-Instruct across differen t datasets. Each subfigure corresp onds to a dataset and rep orts confidence c hanges conditioned on correctness transitions. The Cost of Reasoning in VLMs 25 F Answ er and Reasoning lengths T able 9: A v erage answer and reasoning lengths (num b er of tokens; mean ± std). Answer T okens Answer T okens Reasoning T ok ens Model Dataset No Reasoning Reasoning Reasoning Qwen/Qw en3-VL-32B-Instruct MMMU Pro Vision 1.0 ± 0.0 1.0 ± 0.0 521.9 ± 1742.1 Math Vista 1.8 ± 1.8 1.4 ± 1.1 206.8 ± 842.9 OK-VQA 2.0 ± 1.0 2.2 ± 1.2 47.8 ± 24.7 Oxford Pets 1.7 ± 0.4 1.7 ± 0.4 60.9 ± 11.3 Qwen/Qw en3-VL-8B-Instruct MMMU Pro Vision 1.0 ± 0.0 1.0 ± 0.0 186.0 ± 439.2 Math Vista 1.5 ± 1.5 1.6 ± 1.6 168.3 ± 595.0 OK-VQA 2.0 ± 0.9 2.2 ± 1.1 48.6 ± 16.0 Oxford Pets 3.8 ± 0.5 3.8 ± 0.4 164.0 ± 88.7 Qwen/Qw en3-VL-8B-Thinking MMMU Pro Vision – 1.0 ± 0.0 1879.1 ± 1952.3 Math Vista – 1.4 ± 1.3 1373.4 ± 2317.1 OK-VQA – 2.0 ± 1.0 294.7 ± 407.8 Oxford Pets – 1.8 ± 0.4 127.5 ± 213.6 google/gemma-3-4b-it MMMU Pro Vision 1.0 ± 0.2 1.0 ± 0.3 89.3 ± 45.8 Math Vista 2.2 ± 2.2 1.8 ± 2.2 103.4 ± 110.7 OK-VQA 1.8 ± 0.9 2.0 ± 1.0 40.7 ± 7.9 Oxford Pets 1.8 ± 0.4 1.8 ± 0.4 46.2 ± 7.6 G Rep eated Reasoning and Answ ering. T o b etter understand how the final answer conditions on past p ossible answ ers presen t in the reasoning trace, we designed a controlled to y experiment. W e prompted Qwen3-VL-8B-IT to reason and answer in the standard CoT prompt format, then extended the conv ersation ov er multiple turns, asking the mo del to reason further and answer again while all previous reasoning traces and answers remained in context. W e ran this on a random subset of Math Vista (n=200) and trac ked the sum of the log-probabilities ov er th e answer tokens for eac h instance of answ ering. As shown in Fig. 6, the first-round answer carries a meaningful (non-zero) log-probabilit y . After just one round of conditioning on a prior answer and reasoning trace, subsequent log-probabilities collapse tow ard zero, indicating near-p erfect apparen t confidence. This “confidence collapse” is not earned: the mo del is not b ecoming more accurate across rounds, it is merely b ecoming more committed to rep eating itself. This toy exp erimen t illustrates implicit answer conditioning in its most ex- treme form. Even in standard single-turn CoT prompting, a reasoning trace that builds strongly tow ard a conclusion creates the same conditioning pressure on the final answer tokens, just to a lesser degree. Crucially , the answ er need not app ear verbatim in the reasoning trace for this effect to o ccur, it is sufficient for the reasoning to con verge seman tically on a conclusion. 26 R. W elch et al. Fig. 6: Sequential Confidence T ra jectory . The log-sum-probability of the answ er tokens across m ultiple rounds of reasoning and answ ering. Qw en3-VL-8B-IT on a random subset (n=200) of Math Vista. H Answ er F requency Diagrams (a) Gemma3-4b-IT (b) Qwen3-VL-8b-IT Fig. 7: Reasoning T race Analysis on Incorrect Samples. The frequency that the final answ er is found in the reasoning trace of sampled with the incorrect answer. Higher frequency tends to lead to a greater answer token confidence. The Cost of Reasoning in VLMs 27 I Answ er Masking in Reasoning T races: Math Vista T able 10: Masked-CoT Results on Math Vista. Masking answer mentions par- tially restores uncertaint y ranking, while random masking of an equiv alent num b er of reasoning tokens degrades ranking quality . No-CoT (Acc 62.9) CoT (A cc 72.9) Masked-CoT (Acc 72.9) Random Mask (Acc 72.9) UQ Metho d AUGRC ↓ Spear. ↑ PRR ↑ AUGR C ↓ Sp ear. ↑ PRR ↑ AUGR C ↓ Sp ear. ↑ PRR ↑ AUGR C ↓ Spear. ↑ PRR ↑ MSP 0.142 0.313 0.522 0.130 0.043 0.115 0.129 0.055 0.200 0.147 -0.083 -0.032 MTE 0.143 0.304 0.526 0.134 0.013 0.075 0.119 0.133 0.264 0.144 -0.062 0.010 PPL 0.145 0.289 0.513 0.134 0.017 0.100 0.131 0.038 0.184 0.148 -0.094 -0.050