ALIVE: Awakening LLM Reasoning via Adversarial Learning and Instructive Verbal Evaluation

The quest for expert-level reasoning in Large Language Models (LLMs) has been hampered by a persistent \textit{reward bottleneck}: traditional reinforcement learning (RL) relies on scalar rewards that are \textbf{costly} to scale, \textbf{brittle} across domains, and \textbf{blind} to the underlying logic of a solution. This reliance on external, impoverished signals prevents models from developing a deep, self-contained understanding of reasoning principles. We introduce \textbf{ALIVE} (\emph{Adversarial Learning with Instructive Verbal Evaluation}), a hands-free alignment framework that moves beyond scalar reward optimization toward intrinsic reasoning acquisition. Grounded in the principle of \emph{Cognitive Synergy}, ALIVE unifies problem posing, solving, and judging within a single policy model to internalize the logic of correctness. By coupling adversarial learning with instructive verbal feedback, ALIVE enables models to internalize evaluative criteria directly from raw corpora, effectively transforming external critiques into an endogenous reasoning faculty. Empirical evaluations across mathematical reasoning, code generation, and general logical inference benchmarks demonstrate that ALIVE consistently mitigates reward signal limitations. With identical data and compute, it achieves accuracy gains, markedly improved cross-domain generalization, and higher self-correction rates. These results indicate that the reasoning trinity fosters a self-sustaining trajectory of capability growth, positioning ALIVE as a scalable foundation for general-purpose reasoning alignment without human-in-the-loop supervision.

💡 Research Summary

**
The paper tackles a fundamental limitation of current reasoning‑oriented large language models (LLMs): the “reward bottleneck” inherent in reinforcement‑learning‑based fine‑tuning. Traditional RLHF or RLAIF pipelines rely on scalar rewards that are expensive to obtain, brittle across domains, and blind to the multi‑step logical structure of a solution. To overcome this, the authors propose ALIVE (Adversarial Learning with Instructive Verbal Evaluation), a self‑supervised reinforcement‑learning framework that unifies three cognitive roles—Constructor, Solver, and Reviewer—within a single policy model πθ.

Constructor: Given raw text d, the model masks salient logical spans to create a set of (masked input, ground‑truth) tasks. The masking policy is trained adversarially: it receives a reward proportional to the difficulty it creates for the Solver (1 – accuracy) while being gated by a solvability indicator (accuracy > 0) and regularized by a KL term against a reference policy. This encourages the generation of challenging yet solvable tasks without collapsing into meaningless noise.

Solver: For each constructed query x̃, the model generates multiple candidate solutions ŷ = (z, a) consisting of a reasoning trace z and a final answer a. The Solver’s reward combines a hard binary component r_hard (ExactMatch with the hindsight ground truth) and a dense soft component r_soft derived from the Reviewer’s quality score v ∈