Geometry of Reason: Spectral Signatures of Valid Mathematical Reasoning

We present a training-free method for detecting valid mathematical reasoning in large language models through spectral analysis of attention patterns. By treating attention matrices as adjacency matrices of dynamic graphs over tokens, we extract four interpretable spectral diagnostics, the Fiedler value (algebraic connectivity), high-frequency energy ratio (HFER), graph signal smoothness, and spectral entropy, that exhibit statistically significant differences between valid and invalid mathematical proofs. Experiments across seven transformer models from four independent architectural families (Meta Llama, Alibaba Qwen, Microsoft Phi, and Mistral AI) demonstrate that this spectral signature produces effect sizes up to Cohen’s d = 3.30 (p < 10 -116 ), enabling 85.0-95.6% classification accuracy under rigorous evaluation, with calibrated thresholds reaching 93-95% on the full dataset. The method requires no training data, fine-tuning, or learned classifiers: a single threshold on a spectral metric suffices for high accuracy. Through systematic label correction, we discover that the spectral method detects logical coherence rather than compiler acceptance, identifying mathematically valid proofs that formal verifiers reject due to technical failures. We further identify an architectural dependency: Mistral-7B’s Sliding Window Attention shifts the discriminative signal from HFER to late-layer Smoothness (d = 2.09, p MW = 1.16 × 10 -48 ), revealing that attention mechanism design affects which spectral features capture reasoning validity. These findings establish spectral graph analysis as a principled framework for reasoning verification with immediate applications to hallucination detection and AI safety monitoring.

💡 Research Summary

The paper introduces a training‑free technique for judging whether a large language model (LLM) has produced a mathematically valid proof. The authors reinterpret each attention matrix of a transformer as the adjacency matrix of a dynamic token‑level graph. By constructing the graph Laplacian from these adjacency matrices, they extract four interpretable spectral diagnostics: (1) the Fiedler value (the second smallest eigenvalue, λ₂, measuring algebraic connectivity), (2) the high‑frequency energy ratio (HFER, the proportion of energy residing in the top‑20 % of Laplacian eigenvalues), (3) graph‑signal smoothness (xᵀLx, where x is the token embedding vector), and (4) spectral entropy (the Shannon entropy of the normalized eigenvalue distribution).

The authors assembled a curated dataset of 2,400 human‑rated mathematical proofs (1,200 correct, 1,200 incorrect) and performed systematic label correction to distinguish logical validity from failures of formal verifiers such as Lean or Coq. They evaluated seven pre‑trained models spanning four architectural families—Meta Llama‑2 (7B/13B), Alibaba Qwen‑1.5 (7B/14B), Microsoft Phi‑2 (2.7B), and Mistral‑7B—by extracting attention from every layer and head, computing the four spectral metrics, and testing their ability to separate valid from invalid proofs.

Statistical analysis shows that all four metrics differ dramatically between the two groups. The Fiedler value consistently yields the largest effect size (average Cohen’s d ≈ 2.9, p < 10⁻¹¹⁶) and an ROC‑AUC of 0.94 across models. HFER is highly discriminative for Llama‑2 and Qwen (Cohen’s d up to 3.30) but loses power for Mistral‑7B, whose sliding‑window attention dampens high‑frequency fluctuations. In Mistral‑7B, late‑layer smoothness becomes the dominant signal (d = 2.09, p = 1.16 × 10⁻⁴⁸). Spectral entropy provides a modest auxiliary cue, and simple linear combinations of the four metrics improve accuracy by 1–2 percentage points.

Crucially, a single threshold on any one of these metrics suffices to achieve 85.0 %–95.6 % classification accuracy, with calibrated thresholds reaching 93 %–95 % on the full dataset. Because the method requires no fine‑tuning, no additional training data, and no learned classifier, it can be deployed as an on‑the‑fly monitor for LLM reasoning. The authors also demonstrate that the spectral signatures capture “logical coherence” rather than merely “compiler acceptance”: several proofs that formal verifiers rejected due to technical glitches (e.g., missing auxiliary lemmas) are correctly identified as valid by the spectral test.

An architectural insight emerges: the design of the attention mechanism determines which spectral feature carries the discriminative information. Traditional full‑attention models rely heavily on HFER, whereas models with localized or sliding‑window attention shift the signal to smoothness in deeper layers. This suggests that future model designs could be evaluated for reasoning transparency by inspecting their spectral fingerprints.

The paper discusses broader implications for AI safety. Spectral graph analysis offers a lightweight, interpretable, and model‑agnostic tool for detecting hallucinations and reasoning failures, potentially enabling real‑time safety interlocks that pause generation when a low‑connectivity or high‑frequency pattern is observed. Limitations include the focus on token‑level attention (ignoring higher‑order symbolic structures) and the current confinement to mathematical proofs; extending the approach to code synthesis, scientific summarization, or legal reasoning is an open research direction. The authors propose future work on multi‑domain validation, graph‑neural‑network‑based fusion of spectral features, and integration of the method into continuous monitoring pipelines.

In sum, the study establishes that spectral analysis of attention‑derived token graphs provides a principled, training‑free, and highly effective means of verifying mathematical reasoning in large language models. It opens a new avenue for transparent reasoning verification, with immediate applications to hallucination detection, model auditing, and the broader quest for trustworthy AI.