Interpreting and Controlling LLM Reasoning through Integrated Policy Gradient

Large language models (LLMs) demonstrate strong reasoning abilities in solving complex real-world problems. Yet, the internal mechanisms driving these complex reasoning behaviors remain opaque. Existing interpretability approaches targeting reasoning either identify components (e.g., neurons) correlated with special textual patterns, or rely on human-annotated contrastive pairs to derive control vectors. Consequently, current methods struggle to precisely localize complex reasoning mechanisms or capture sequential influence from model internal workings to the reasoning outputs. In this paper, built on outcome-oriented and sequential-influence-aware principles, we focus on identifying components that have sequential contribution to reasoning behavior where outcomes are cumulated by long-range effects. We propose Integrated Policy Gradient (IPG), a novel framework that attributes reasoning behaviors to model’s inner components by propagating compound outcome-based signals such as post reasoning accuracy backward through model inference trajectories. Empirical evaluations demonstrate that our approach achieves more precise localization and enables reliable modulation of reasoning behaviors (e.g., reasoning capability, reasoning strength) across diverse reasoning models.

💡 Research Summary

The paper addresses the opacity of reasoning processes in large language models (LLMs) by proposing a novel, training‑free interpretability and control framework called Integrated Policy Gradient (IPG). Existing interpretability methods for reasoning either correlate internal units (neurons, sparse auto‑encoder features) with surface text patterns or rely on human‑crafted contrastive pairs to derive control vectors. Both approaches suffer from two major drawbacks: they focus on co‑occurrence rather than causal contribution, and they capture only short‑term effects, failing to account for the cumulative, multi‑step nature of reasoning.

IPG is built on two guiding principles: (i) outcome‑oriented evaluation, meaning that the importance of an internal component is measured by its contribution to a downstream reasoning outcome (e.g., answer correctness, number of generated tokens); and (ii) sequential‑influence awareness, acknowledging that reasoning unfolds over a long horizon and that each component’s effect may be distributed across many steps.

Methodologically, IPG extends the classic policy‑gradient algorithm from the parameter space to the representation space of the model. For a given reasoning trajectory τ = (a₁,…,a_T) generated by the LLM policy π_θ, a signal function J(·) (binary reward for correctness or token‑count for strength) provides a scalar feedback. The gradient of J with respect to an internal component hₜ (which can be a hidden‑state neuron or an SAE‑derived feature) is computed as an expectation over trajectories of ∂ log π_θ(aₜ|sₜ, hₜ) · A_π(sₜ, aₜ), where A_π is an advantage estimator reflecting long‑term benefit.

To overcome the noisiness of pointwise gradients, IPG incorporates a path‑integral (Integrated Gradients) step: each component’s activation is interpolated from a baseline h′ₜ (typically zero) to its actual value hₜ, and the policy‑gradient is accumulated along this line. The resulting attribution score, IPG(i; x), integrates the influence of component i across all time steps and across the entire interpolation path, yielding a baseline‑aware, globally consistent importance measure. Scores are computed per sample and aggregated (e.g., mean over a supporting dataset) to identify the top‑p most influential components.

Experiments are conducted on several open‑source LLMs (LLaMA‑2, Qwen‑1.5) and on fine‑tuned reasoning‑enhanced variants. Two behavioral metrics are examined: reasoning capability (binary correctness) and reasoning strength (trajectory length). After identifying the top components via IPG, the authors intervene by scaling the selected neurons or SAE features by a factor γ (γ > 1 to enhance, γ < 1 to suppress). Results show that enhancing the top components raises accuracy by up to 12 percentage points, while suppressing them reduces accuracy by up to 9 percentage points. Similar trends are observed for reasoning strength, confirming that IPG can reliably steer both dimensions. Notably, components discovered in a base model transfer effectively to its fine‑tuned counterpart, demonstrating cross‑model transferability and reducing the need for repeated interpretation.

The paper also provides ablations: (a) comparing raw policy gradients versus IPG’s integrated version (the latter yields smoother, more stable attributions); (b) varying the baseline and number of interpolation steps; and (c) evaluating the effect of different advantage estimators. Limitations are acknowledged: policy‑gradient estimates can be noisy with small sample sizes, SAE training adds computational overhead, and the current binary reward formulation may not capture nuanced reasoning qualities such as logical consistency.

In summary, Integrated Policy Gradient offers a principled way to attribute long‑horizon reasoning outcomes to specific internal components of LLMs, surpassing prior text‑pattern and contrastive‑vector methods in both precision and controllability. By enabling fine‑grained, outcome‑driven interventions without modifying model weights, IPG opens new avenues for safety‑critical applications, model debugging, and targeted capability enhancement. Future work is suggested on multi‑objective reward design, more efficient path‑integration approximations, and extending the framework to multimodal or instruction‑tuned models.