Evaluating Chain-of-Thought Reasoning through Reusability and Verifiability

In multi-agent IR pipelines for tasks such as search and ranking, LLM-based agents exchange intermediate reasoning in terms of Chain-of-Thought (CoT) with each other. Current CoT evaluation narrowly focuses on target task accuracy. However, this metric fails to assess the quality or utility of the reasoning process itself. To address this limitation, we introduce two novel measures: reusability and verifiability. We decouple CoT generation from execution using a Thinker-Executor framework. Reusability measures how easily an Executor can reuse the Thinker’s CoT. Verifiability measures how frequently an Executor can match the Thinker’s answer using the CoT. We evaluated four Thinker models against a committee of ten Executor models across five benchmarks. Our results reveal that reusability and verifiability do not correlate with standard accuracy, exposing a blind spot in current accuracy-based leaderboards for reasoning capability. Surprisingly, we find that CoTs from specialized reasoning models are not consistently more reusable or verifiable than those from general-purpose LLMs like Llama and Gemma.

💡 Research Summary

The paper tackles a fundamental blind spot in the evaluation of chain‑of‑thought (CoT) reasoning generated by large language models (LLMs). While most recent work measures the quality of CoT indirectly by looking at the final answer accuracy on benchmark tasks, this approach cannot distinguish whether a model arrived at the answer through genuine step‑by‑step reasoning or by memorization, brute‑force search, or other shortcuts. To address this, the authors propose two orthogonal metrics—reusability and verifiability—that assess the intrinsic utility of the reasoning trace itself, independent of the end‑task outcome.

Thinker‑Executor Framework
The authors introduce a modular “Thinker‑Executor” paradigm that explicitly separates CoT generation (Thinker) from its execution (Executor). The Thinker receives a question, produces its own answer together with a CoT. The Executor then receives the same question augmented with the Thinker’s CoT and attempts to answer. By varying the quality of the CoT (correct vs. deliberately corrupted) and observing how the Executor’s answer changes, the two metrics are defined as follows:

• Reusability measures the proportion of cases where the Executor’s answer flips in the expected direction when supplied with a Thinker’s CoT. If the Executor originally answers incorrectly, a correct CoT should turn the answer correct; conversely, if the Executor originally answers correctly, a corrupted CoT should cause it to err. The final score is the sum of these two successful flips divided by the total number of questions for which the Thinker’s own answer is correct. High reusability indicates that the CoT is model‑agnostic and persuasive enough that other agents can follow it without relying on their own internal reasoning.

• Verifiability captures consistency: for a given CoT, how often do different Executors produce the same final answer as the Thinker? It is simply the percentage of questions where the Executor’s answer (when using the Thinker’s CoT) matches the Thinker’s answer. High verifiability signals that the reasoning trace is unambiguous and can be interpreted uniformly across heterogeneous models.

Experimental Setup
Four Thinker models are evaluated: two general‑purpose LLMs (Gemma‑3‑27B and Llama‑3.1‑8B) and two specialized reasoning models (DeepSeek‑R1‑14B and Phi‑4‑Reasoning‑14B). Ten Executor models, ranging from 360 M to 3 B parameters, form three committees: Weak (five smallest models), Strong (five largest models), and Full (all ten). The authors test across five reasoning‑oriented benchmarks: GSM8K and SV‑AMP (math), StrategyQA (multi‑step logic), ARC‑Challenge (science), and CommonSenseQA (commonsense). All experiments run on a single NVIDIA A100 GPU.

Key Findings

1. Low Correlation with Accuracy – Kendall’s τ between accuracy and either reusability or verifiability is consistently low (often near zero or even negative). A model that tops the accuracy leaderboard does not necessarily produce CoTs that are reusable or verifiable. For example, on GSM8K DeepSeek and Gemma‑3 achieve the highest accuracy (94 % and 93 %), yet Phi‑4‑Reasoning attains the highest reusability (83.53 %) and verifiability (88.36 %).

2. Specialized Reasoning Models Not Universally Superior – The two dedicated reasoning models do not dominate the general‑purpose models across the new metrics. Phi‑4‑Reasoning often scores best, but DeepSeek‑R1 sometimes lags behind Gemma or Llama, especially on the SV‑AMP dataset where Llama shows the highest reusability despite poor accuracy.

3. Committee Strength Affects Absolute Scores but Not Rankings – Strong committees yield higher absolute reusability and verifiability scores than Weak committees (e.g., Strong > Full > Weak). However, the relative ordering of Thinker models remains stable: Kendall’s τ for reusability between Strong and Full committees is 1.0, and for verifiability it is 0.8. This suggests that while score magnitudes are sensitive to the capabilities of the Executors, the comparative quality of Thinker CoTs is robust when at least a few strong Executors are present.

4. Interpretation of Metrics – Reusability is likened to persuasive speech: a high score means the CoT can convince another model to adopt its line of reasoning, which is valuable for collaborative multi‑agent systems but also opens a vector for adversarial attacks (e.g., feeding a malicious CoT to mislead downstream agents). Verifiability resembles a legal contract: a high score indicates the reasoning is unambiguous and yields the same conclusion regardless of who interprets it, which is crucial for transparency and auditability.

Limitations and Future Work
The authors acknowledge several constraints: (a) the generation of corrupted CoTs relies on handcrafted prompts, limiting scalability and introducing human bias; (b) the Executor pool consists only of relatively small models, leaving open the question of how well the metrics transfer to very large LLMs (30 B +); (c) the practical impact of reusability and verifiability on user trust, system safety, or downstream task performance remains to be quantified. Future directions include extending the framework to other domains (code generation, medical diagnosis), integrating automated CoT validators to provide real‑time feedback, and exploring how these metrics can be incorporated into reward models or alignment pipelines.

Conclusion
By decoupling reasoning generation from execution and introducing reusability and verifiability, the paper provides a concrete, quantitative lens for assessing the process of LLM reasoning rather than just its outcome. The empirical results demonstrate that current accuracy‑centric leaderboards overlook substantial aspects of reasoning quality, and that both general‑purpose and specialized models can produce CoTs with varying degrees of utility. This work lays groundwork for more nuanced evaluation regimes, safer multi‑agent collaborations, and ultimately more trustworthy AI systems that can not only give the right answer but also explain it in a way that others can reliably follow and verify.