Can Aha Moments Be Fake? Identifying True and Decorative Thinking Steps in Chain-of-Thought

Large language models can generate long chain-of-thought (CoT) reasoning, but it remains unclear whether the verbalized steps reflect the models’ internal thinking. In this work, we propose a True Thinking Score (TTS) to quantify the causal contribution of each step in CoT to the model’s final prediction. Our experiments show that LLMs often interleave between true-thinking steps (which are genuinely used to compute the final output) and decorative-thinking steps (which give the appearance of reasoning but have minimal causal influence). We reveal that only a small subset of the total reasoning steps causally drive the model’s prediction: e.g., on AIME, only an average of 2.3% of reasoning steps in CoT have a TTS >= 0.7 (range: 0-1) for Qwen-2.5. Furthermore, we find that LLMs can be steered to internally follow or disregard specific steps in their verbalized CoT using the identified TrueThinking direction. We highlight that self-verification steps in CoT (i.e., aha moments) can be decorative, while steering along the TrueThinking direction can force internal reasoning over these steps. Overall, our work reveals that LLMs often verbalize reasoning steps without performing them internally, challenging the efficiency of LLM reasoning and the trustworthiness of CoT.

💡 Research Summary

The paper tackles a fundamental question about large language models (LLMs): does the chain‑of‑thought (CoT) that a model generates truly reflect the reasoning steps it performs internally? To answer this, the authors introduce the True Thinking Score (TTS), a metric that quantifies the causal contribution of each verbalized step to the model’s final prediction. TTS is built from two complementary interventions. The necessity test (ATE_nec) measures how the model’s confidence in the answer changes when a specific step s is perturbed while keeping all preceding steps C intact. The sufficiency test (ATE_suf) measures the same change when the context C is corrupted but the step s remains. By averaging the absolute values of these two effects, TTS captures both “AND‑type” contributions (where a step must be combined with earlier steps) and “OR‑type” contributions (where a step alone can determine the answer).

The authors evaluate TTS on several state‑of‑the‑art LLMs—including Qwen‑2.5, GPT‑4, and Llama‑2—across a variety of benchmarks such as AIME, GSM‑8K, and synthetic self‑verification examples. Their findings are striking: only about 2 % of the total CoT steps achieve a TTS ≥ 0.7, meaning that the overwhelming majority of the generated reasoning is “decorative” and has negligible causal impact on the final answer. In particular, the often‑cited “aha moments” where the model says “Wait, let’s re‑evaluate…” are usually decorative; random perturbations of the numbers preceding these moments rarely affect the answer.

Beyond measurement, the paper uncovers a mechanistic lever: a latent‑space direction termed the “TrueThinking direction.” By moving the hidden representation of a step along this vector, the authors can increase or decrease the step’s internal influence. Positive shifts raise the step’s TTS and cause the model to actually use the step in its computation; negative shifts suppress it, effectively turning the step into a pure post‑hoc explanation. Using this steering technique, the authors force the model to internally reason over previously decorative self‑verification steps, demonstrating direct control over the model’s internal reasoning process.

The work has several important implications. First, it challenges the common practice of treating CoT as a faithful “scratch‑pad” for monitoring model behavior or safety; reliance on CoT alone may be misleading. Second, the TTS framework provides a fine‑grained diagnostic tool for researchers to assess and improve the faithfulness of generated reasoning. Third, the TrueThinking direction opens a new avenue for steering LLMs toward more transparent and efficient reasoning, potentially reducing unnecessary computational overhead from decorative steps.

Limitations include the dependence of TTS on probability estimates, which may be noisy for low‑confidence models, and the need to verify whether the TrueThinking direction generalizes across architectures and tasks. Future work could integrate TTS into training objectives to reward genuinely useful reasoning steps and penalize decorative ones, as well as explore broader safety applications.

In summary, the paper delivers a rigorous causal analysis of CoT, introduces a robust metric (TTS) and a controllable latent‑space steering vector (TrueThinking direction), and demonstrates that most CoT steps are superficial. This advances our understanding of LLM internal reasoning, raises caution about over‑trusting verbalized explanations, and provides concrete tools for building more trustworthy, efficient, and controllable language models.