Theorem 1 ((Informal) Failure-driven training minimizes the expected loss.). *Let $`\nabla {\mathcal{L}}^*(\theta)`$ be the ideal target graident, $`\nabla {\mathcal{L}}_{1}(\theta)`$ be the stage-2 gradient under the synthetic distribution $`P_{1}`$, and $`\nabla {\mathcal{L}}_0(\theta)`$ be the gradient under the original distribution $`P_0`$. Assuming the gradient is normalized such that $`\|\nabla {\mathcal{L}}_1(\theta)\|=\|\nabla {\mathcal{L}}_0(\theta)\|`$ and $`Q\approx P*`$, the failure-driven construction of $`P_1`$ makes the gradients better aligned with the target gradient, i.e.,

\begin{equation}
\langle \nabla {\mathcal{L}}^*(\theta), \nabla{\mathcal{L}}_1(\theta)\rangle
\ge
\langle \nabla {\mathcal{L}}^*(\theta), \nabla {\mathcal{L}}_0(\theta)\rangle,
\end{equation}

Then, for sufficiently small learning rate $`\eta`$, one stage-2 optimization step using $`P_{1}`$ yields no larger target risk than using $`P_0`$:

\begin{equation}
{\mathcal{L}}^*(\theta-\eta \nabla  {\mathcal{L}}_1(\theta)) \le {\mathcal{L}}^*(\theta-\eta \nabla {\mathcal{L}}_0(\theta)).
\end{equation}
```*

</div>

</div>

We left the proof and detailed derivation
in <a href="#app:grad_alignment" data-reference-type="ref+label"
data-reference="app:grad_alignment">9.2</a>.

### Extension to Reinforcement learning

Our framework is derived under the purpose of dealing the mismatch
between training and target distribution, thus it can be naturally
extended to design a second stage training under reinforcement learning
via verified reward (RLVR). We do not elaborate the theoretical proof in
the main text, as it follows a similar structure as failure-driven SFT,
but only show the algorithm here. The detailed derivation can be found
in <a href="#apdx: ext_RL" data-reference-type="ref+label"
data-reference="apdx: ext_RL">9.3</a>

#### Stage-2 RLVR.

We first sample the prompts from the marginal distribution $`P_1(x)`$.
Then, for a prompt $`x`$, the model samples an output
$`y\sim \pi_\theta(\cdot\mid x)`$ and receives a verified reward as the
advantage function $`A`$, where
``` math
\begin{equation}
\label{eq:rl-binary}
A(x,y)\in\{0,1\},
\quad
A(x,y)=1 \;\;\text{iff the final answer in $y$ is verified correct}.
\end{equation}

The vanilla RL objective is

\begin{equation}
\label{eq:stage2-rl}
{\mathcal{L}}(\theta) = \mathbb{E}_{x\sim P_{1}(x)}\;
\mathbb{E}_{y\sim \pi_\theta(\cdot\mid x)}\big[A(x,y)\big].
\end{equation}

Practically, we test our model using GRPO  and DAPO , which we defer the introduction in 4.2.4.

Engineering Implementation

We then introduce our engineering implementation. The central tenet of second-stage training and data usage is to identify the underestimated region from the failure cases and remediate them using external, high-fidelity knowledge sources. This is done through three steps: 1) knowledge base construction, 2) knowledge retrieval engine, and 3) curated question-answer (QA) pair generation. The overall pipeline is illustrated in 7. Notably, data employed in the second stage is generated automatically without any human annotation.

Knowledge Base Construction

We curated an internal multi-source PDF corpus encompassing academic papers, technical monographs, authoritative textbooks, and technical reports. To preserve both structural fidelity and semantic integrity, we employed LogicsParsing(),a proprietary, high-precision PDF parsing tool, to convert raw documents into structured HTML format. Subsequently, we applied a multi-dimensional quality filtering mechanism:

• A text quality model evaluated linguistic fluency;

• A usefulness model quantified information richness;

• A fine-grained subject classification model assigned domain-specific labels.

Only documents exhibiting high fluency, high knowledge density, and unambiguous domain specificity were retained, culminating in a high-quality, structurally coherent knowledge base.

Knowledge Retrieval

We first evaluate the SFT model on multiple established STEM benchmarks, including AIME2025 , MATH500, MMLU  that works as the gold-standard distribution $`Q`$. We then collect questions answered incorrectly as “knowledge deficiency” samples. These erroneous instances are then encoded into dense vector representations using the Qwen3-8B-Embed⁶. Concurrently, every document in the knowledge base is embedded using the same model. Leveraging a vector retrieval algorithm, we retrieve the top-30 semantically most relevant documents for each incorrect question, ensuring high topical, conceptual, or methodological alignment between the query and retrieved content.

Data Synthesis

Utilizing the DeepSeek-R1  as the synthesis kernel $`G((x^\prime, y^\prime)\mid d)`$, we automatically generate two Query-Response pairs per retrieved document, each requiring deep reasoning based on the document’s core knowledge points and logical structure. To guarantee answer accuracy and consistency, a dual-verification mechanism is implemented to serve as the acceptance filter:

• In the first pass, a question and a preliminary answer are generated jointly;

• In the second pass, the model is prompted with the question alone to generate an independent answer;

• Only samples yielding identical answers across both passes are retained as high-confidence training instances.

For inconsistent cases, we aggregate additional model responses and applied a majority-voting mechanism to determine a consensus answer; These samples are incorporated into the training set as “hard examples.” Furthermore, the length of the Chain-of-Thought (CoT) reasoning trace generate during the second answer synthesis serves as a proxy indicator for difficulty, as longer CoT sequences typically signify more complex reasoning steps and greater challenge. This criterion facilitates the selection of high-difficulty samples, ensuring the training data not only possesses high reliability but also effectively stimulates the model’s potential for advanced reasoning. Through this rigorous pipeline, we ultimately synthesize a dataset comprising approximately 30K high-quality, high-difficulty, knowledge-aligned QA pairs. This dataset is subsequently used to train the SFT model, resulting in substantial improvements in robustness and generalization on complex reasoning tasks.

Reinforcement Learning Implementation

We test GRPO  and DAPO  for our framework, which defines the policy ratio $`r_{i, t}`$ and the renormalized advantage function $`\hat{A}_{i}`$ as,

\begin{equation}
r_{i, t}(\theta)=\frac{\pi_\theta\left(y_i^t \mid x, y_i^{<t}\right)}{\pi_{\theta_{\text {old }}}\left(y_{i}^t \mid x, y_{i}^{<t}\right)}, \quad \hat{A}_{i}=\frac{A_i-\operatorname{mean}\left(\left\{A_i\right\}_{i=1}^G\right)}{\operatorname{std}\left(\left\{A_i\right\}_{i=1}^G\right)} .
\end{equation}

We adopt the clip-higher strategy alongside batch-level reward normalization. Specifically, for each $`x\sim P_1(x)`$, we sample multiple $`\{y_i\}_{i=1}^G`$, and defines the

\begin{equation}
\label{eq: DAPO}
    R(x,y) =\left[\frac{1}{\Sigma_{i=1}^{G}|y_i|}\sum_{i=1}^{G}\sum_{t=1}^{|y_i|}\min(r_{i, t}(\theta)\hat{A}_{i}, \text{clip}(1+\varepsilon, 1-\varepsilon, r_{i, t}(\theta))\hat{A}_{i})\right]
\end{equation}

In addition to the answer correctness reward as in [eq:rl-binary], we add format compliance and reasoning-length reward to the RL.

Answer Correctness.

We use the open-source framework ROLL  to compute verified rewards. For math problems, we use math-verify to compare model outputs against ground-truth answers and assign a binary reward as advantage $`A_i\in\{0,1\}`$. For multiple-choice questions, we extract the option in $`\texttt{\textbackslash boxed}\{...\}`$ and match it to the ground truth. Following , we apply dynamic sampling to filter problematic training instances by (i) dropping over-length responses and (ii) excluding prompts whose response-group mean scores fall outside a preset range. Hyperparameters are summarized in Table 12.

Length and Repetition Aware Reward.

We observe that even when the final answer is incorrect, longer and more explicit CoT traces often facilitate self-correction and gradual convergence to the correct solution. Accordingly, we encourage deeper reasoning by assigning a length-based reward: for incorrect answers, the reward increases monotonically with the CoT length; for correct answers, we grant the maximum length reward by default to avoid discouraging concise yet correct reasoning. We also find that CoT generations can be redundant. To improve diversity and information density, we incorporate an $`n`$-gram–based repetition penalty into the reward.

Formally, let $`A\in\{0,1\}`$ indicate answer correctness, $`\ell`$ be the reasoning-trace length, and $`\ell_{\min},\ell_{\max}`$ be length bounds. We define the normalized length score

s(T,A)=
\begin{cases}
1, & A=1 \ \text{or}\ T\ge T_{\max},\\
0, & A=0 \ \text{and}\ T\le T_{\min},\\
\dfrac{T-T_{\min}}{T_{\max}-T_{\min}}, & \text{otherwise}.
\end{cases}

The final length-aware reward is $`r_{\text{len}} = s(T,A)\cdot \rho,`$ where $`\rho\in(0,1]`$ is a penalty factor computed from the $`n`$-gram repetition rate of the generated text.

Experiments

With systematic experiments, we show that our framework not only yields significant gains in reasoning accuracy on STEM benchmarks but also establishes a novel, reproducible, transferable, and highly efficient paradigm for data construction in the post-training phase.

General Evaluation

Benchmarks

Models are evaluated on representative reasoning benchmarks spanning mathematics and broader STEM disciplines. For mathematical reasoning, we assess performance on AIME2024 , AIME2025, HMMT_Feb_2025, BRUMO2025, BeyondAIME, and MATH-500. For remaining STEM domains, we adopt GPQA-Diamond , R-Bench , MMLU-Pro , and CMMLU as benchmarks. Particularly for MMLU-Pro and CMMLU, we select STEM-related subsets to specifically evaluate the model’s reasoning capabilities in STEM fields, as detailed in 7.4.

Evaluation Setting

To ensure statistical robustness and mitigate sampling variability, we conduct $`N`$ independent generations for each test instance using zero-shot evaluation, adhering to the generation configurations detailed in 10 by default. For reasoning-intensive benchmarks with a limited number of test instances, such as AIME2024, AIME2025, BeyondAIME, HMMT_Feb_2025, and GPQA-Diamond, we perform $`16`$ generations per case. In contrast, for elementary benchmarks or those with a larger number of test instances, we use up to $`4`$ generations per instance. Consequently, we report Pass@1 as the primary evaluation metric, with additional Pass@K (also referred to as Best@N) and majority-vote accuracy (Majority@N) as supplementary metrics. Further details of evaluation settings are described in 8.1.

General Evaluation Results

As shown in [tab:3,tab:4], Logics-STEM-8B-RL significantly outperforms other reasoning models of comparable size from the community, scoring 90.42% on AIME2024, 87.08% on AIME2025, 74.79% on HMMT2025, 62.5% on BeyondAIME and 73.93% on GPQA-Diamond. The consistently strong performance across diverse benchmarks further evidences the robust domain-generalization capacity of the proposed data curation strategy, demonstrating the effectiveness of our SFT and RLVR strategies.

Notably, Logics-STEM-8B-SFT achieves reasoning performance comparable to state-of-the-art open-source reinforcement learning models of similar scale, such as Klear-Reasoner-8B. This finding demonstrates our ability to fully leverage high-quality reasoning datasets and further suggests that, for models of small size, knowledge distillation through SFT can be as effective as RL-centric post-training.

Moreover, as evidenced by the substantial improvements in the Majority@N on competition-level benchmarks shown in 3, the application of RLVR after supervised fine-tuning (SFT) significantly enhances model reasoning ability by further concentrating the probability distribution around correct answers, even when SFT has already established a strong baseline.

Ablation Study of Data

Sampling Strategy

As explained in 3.2, further sampling over the massive outputs of the curation pipeline is essential. Therefore we conduct a series of ablation experiments to identify the most effective sampling strategy. Specifically, for each sampling strategy, we uniformly sample 200K records to construct the training set and fine-tune the model for one epoch.

Stratified Sampling.

Length-based sampling has been demonstrated to be an effective sampling method ; nevertheless, our experiments demonstrate that its exclusive use degrades the model’s elementary reasoning and generalization capacities, as reported in 4. the ablation model trained solely with length-based sampling incurs significant performance drops on MATH500, MMLU-Pro-Math, and GPQA-Diamond, despite notable gains on AIME2024, AIME2025, and HMMT2025.

Difficulty-Based Sampling.

Another set of ablation experiments focuses on variants of stratified sampling. We explore sampling based on the token length of response generated by the teacher model and the annotated education level of the question. The results in Table 4 demonstrate that the former serves as a natural and more effective proxy for question difficulty.

We therefore adopt length-based stratified sampling to mitigate catastrophic forgetting of elementary reasoning capabilities, while scaling the training set to strengthen advanced reasoning capabilities.

Source of failure-driven Synthetic Data

To investigate how different types of benchmark-recall document synthesis affect second-stage training, we construct training data using two representative benchmarks: (i) Scientific QA (e.g., GPQA), (ii) Mathematical reasoning (e.g., Math500, AIME2024/2025, and HMMT). Following a unified data construction pipeline, we generate second-stage training samples from incorrectly answered questions in these benchmarks and train models independently. Experimental results demonstrate that training data synthesized from GPQA errors not only significantly improves GPQA evaluation performance but also delivers notable generalization gains across mathematical reasoning benchmarks (e.g., AIME, HMMT). In contrast, training data derived solely from mathematical benchmarks yields limited improvement in overall STEM capabilities.

This finding suggests that the complex reasoning patterns inherent in scientific QA tasks may possess stronger cross-domain transferability, making them more valuable for enhancing general reasoning abilities.

Ablation Study of Algorithms

In this section, we examine how different settings and modules influence the RLVR training process. We present ablation experiments with detailed results and demonstrate how they inform the design of our final training recipe.

RLVR Algorithm

Our experiments cover different loss functions, including Group Relative Policy Optimization (GRPO) (), Decoupled Clip and Dynamic Sampling Policy Optimization (DAPO) ().

We first conduct ablation studies on Qwen2.5-7B-Instruct using the DAPO-Math-17K dataset and find that employing the DAPO objective in conjunction with Reinforce++ as the advantage estimator yields more stable and consistent improvements. As shown in 6, an early ablation indicates that the DAPO loss function outperforms the GRPO Loss on our SFT model. Therefore, the DAPO objective coupled with Reinforce++ as the advantage estimator is adopted as the final algorithm.

Dynamic Sampling

Following , we take advantage of dynamic data sampling strategy in our training pipeline. Specifically, we first drop the responses that are longer than our default maximum length, it is called overlong length filter. Also, for the difficulty filter, for each prompt, we will calculate the average score within the group of all its responses, if it is lower or greater than our set threshold, the data will not be involved in the calculation of loss function.

In this experiment, we conduct two groups of trials: one using our original setting (with both the overlong-length and difficulty filters enabled), and the other with both filters disabled. We set the lower and upper thresholds for the average response score to 0.1 and 0.95, respectively.

As we can observe in 8, the group without a filter tends to collapse in both the training accuracy and response length in an early stage.

Our analysis indicates that response groups with average scores that are either too high or too low yield near-zero advantage values after normalization, which in turn results in negligible or zero gradient updates. Intuitively, such samples are either too “easy” or too “hard” for the current model and provide little useful signal for learning. Additionally, without the overlong-length filter, responses exceeding the maximum allowed length are truncated, making it difficult to extract the final output and leading to a short response length. Consequently, both the overlong-length filter and the difficulty filter are essential to ensure stable and effective training.

Length-based Reward and Repetition Penalty

Experimental results demonstrate that incorporating the reasoning-length reward and repetition penalty mechanisms yields significant performance improvements ranging from 2 to 3 percentage points on benchmark datasets, including AIME2025, HMMT, and CMMLU. Consistent, albeit modest, gains are also observed across all other evaluated benchmarks.

The Generalizability of Data Engine

Generalizability of Logics-STEM-SFT-Dataset to Larger-scaled Models

In addition to the superior performance achieved by the model fine-tuned from Qwen3-8B (as presented in 5.1.3), we also experiment with several other models as initial models during the supervised fine-tuning (SFT) stages. All of these experiments further confirm the effectiveness of our data engine, as shown in 8.

Firstly, we fine-tune Qwen3-8B-Base with the identical Logics-STEM-SFT-2.2M dataset. The resulting model performs on par with, though marginally weaker than, Logics-STEM-8B-SFT while substantially surpassing Qwen3-8B. These gains demonstrate that our dataset is sufficiently diverse and of high quality to support full-scale supervised fine-tuning (SFT) in the STEM domain.

Additionally, we evaluate the scalability of our approach by fine-tuning Qwen3-32B. Remarkably, the resulting model, Logics-STEM-32B-SFT, still exhibits substantial improvements in reasoning performance, further corroborating the high quality of our dataset.

Generalizability of Failure-Driven Data

In addition to the RLVR experiments, we conduct continual SFT experiments using failure-driven synthetic data to assess the universal effectiveness of the data across different post-training paradigms. Analogous to the RLVR training setup, we conduct experiments on Logics-8B-SFT, with training hyperparameters detailed in the Appendix. To mitigate catastrophic forgetting, we experimentally construct mixtures of 34K failure-driven synthetic data and existing SFT training data at varying ratios. Notably, under an approximately 1:1 mixture of failure-driven synthetic data and existing SFT data, the model achieves improvements comparable to those obtained with the RLVR approach, as presented in 9 .

Scale Inference Budget

Due to different evaluation settings in officially reported results that limit direct comparisons across models, we systematically assess the effect of inference budget by independently evaluating each model under varying context settings.

As illustrated in [tab:3,tab:4], scaling up inference budget from 32K context to 64K context leads to significant improvements on competition-level benchmarks such as AIME2024, AIME2025, and HMMT2025, whereas improvements on elementary benchmarks remain modest. The observation is consistent with the intuition that more challenging problems require deeper deliberation. Moreover, reasoning-oriented models typically benefit more from the additional inference budget than the vanilla Qwen3-8B baseline.

Unsuccessful Attempts

Numeric-Answer-Only RLVR

In an experiment, we apply reinforcement learning on DAPO-Math-17k, a dataset that exclusively contains numeric verifiable answers. We observe a marked deterioration in the model’s instruction-following ability, specifically in its capacity to respond with option letters as instructed, which leads to significant performance degradation on GPQA-Diamond, R-Bench, and MMLU-Pro-STEM. This observation explains the poor performance of Klear-Reasoner-8B on these benchmarks (See 2).

Entropy Loss in RLVR

In our ablation study, we compare training objectives with and without an entropy loss term, and we also explore adaptive entropy control following the approach of , in which we stick to their hyperparameter setting. Our experiments show that either incorporating a naive entropy loss or implementing adaptive entropy loss requires careful hyperparameter tuning and is prone to causing entropy explosion, as illustrated in 9. Consequently, we exclude this term from our final training recipe.

Scale Failure-Driven Synthetic Data

We further investigate scaling the volume of failure-driven synthetic data by relaxing the relevance threshold to retrieve more educational documents associated with each failure case. However, as detailed in [{appendix: scale failure-driven}], this actually degrades the model’s performance. We hypothesize that the additional documents are less strongly related to the original failures, thereby breaking the expected scaling behavior.

Conclusion

This technical report introduces Logics-STEM, a state-of-the-art reasoning model specifically designed for STEM-related domains. Through proposed data curation pipeline combined with the difficulty-based stratified sampling strategy, we effectively select high-quality reasoning data from both open-access sources and synthetic datasets, thereby ensuring robust supervised fine-tuning. The further integration of failure-driven post-training and complementary dynamic sampling leads to substantial improvements in reasoning performance. For future work, we plan to further investigate reinforcement learning techniques, scale the model to larger architectures, and extend its capabilities to a broader range of tasks, including coding and tool-integrated reasoning.

More about Data

Source of Data

We collect questions from the following highly regarded and frequently cited datasets: NuminaMath1.5 (), OpenThoughts3 (), Mixture-of-Thoughts (), AceReason-1.1-SFT (), AceReason-Math (), OpenScienceReasoning-2 ⁷, OpenMathReasoning (), Llama-Nemotron-Post-Training-Dataset (), DLR-Web (), DLR-Book (), Skywork-OR1-RL-Data (), NaturalReasoning (), DeepMath-103K (), DAPO-Math-17k (), TheoremQA (), JEEBench (), GPQA-Main (), GSM8K (), AIME ⁸, AMC ⁹, s1-teasers (), s1-probs (), openaimath ().

Annotation

Generation Configuration

We adopt the recommended configuration parameters provided by Qwen team for both response distillation and evaluation inference, as detailed in 10.

STEM Benchmarks

More about Evaluation

Setting

Zero-shot Evaluation

To ensure consistency, we employ a uniform zero-shot evaluation protocol across all benchmarks, where the model is instructed to enclose its final answer within $`\backslash`$boxed{} in response to each question.
For benchmarks featuring multiple-choice questions, such as GPQA-Diamond, MMLU-Pro-STEM, CMMLU-STEM, we prompt the model with the following template:

For other benchmarks, the following template is adopted:

Answer Verification

We extract the answer from the \boxed{} expression of the model’s response and evaluate its correctness by comparing it with the ground-truth answer using the math-verify library.

Evaluation Metrics

For each test instance, we calculate three metrics as following:

• Pass@1 is defined as the average accuracy across $`N`$ rollout samples.

• Best@N is assigned a value of 1 if any of the $`N`$ rollout samples matches the gold answer, and 0 otherwise.

• Majority@N is assigned a value of 1 if the majority answer among the $`N`$ rollout samples matches the gold answer, and 0 otherwise.

Consequently, we report the evaluation result of each benchmark as the average of the evaluation metrics computed over all test instances.

Training Setting

Hyperparamaters

Hyperparameters we use are listed in [tab:Hyperparam for SFT,tab: hyperparam for rlvr,tab:Hyperparam for Continual SFT].

Supplementaries for Ablation Studies in RLVR

Adaptive Entropy Control

We experiment with various entropy-based strategies but failed to achieve a stable entropy loss or consistent improvements in accuracy while training, as shown in 9.

Scale Failure-Driven Synthetic Data

As described in 5.6, the volume of failure-driven synthetic data is scaled to 150K. Fine-tuning the model on a 1:1 mixture of these 150 K synthetic instances and 150 K existing SFT examples yields the results reported in 14, where we observe gains only on BeyondAIME and GPQA-Diamond accompanied by performance drops on all other benchmarks, despite the increased number of training steps.

Theoretical Results for Failure-Driven Post-training

The Resources of Failure Modes

Minizing the loss on $`P_0`$ may not minimize the target risk on $`P^*`$.

Consider a model trained by empirical risk minimization (ERM) on the stage-1 training distribution $`P_0`$:

\begin{equation}
\label{eq:erm_p0}
\hat{\theta}_0=\arg\min_{\theta}\ \mathbb{E}_{(x,y)\sim P_0}\big[\ell_\theta (x,y)\big].
\end{equation}

However, our real goal is to minimize the risk under the (unknown) target distribution $`P^*`$:

\begin{equation}
\label{eq:erm_pstar}
\theta^*=\arg\min_{\theta}\ \mathbb{E}_{(x,y)\sim P^*}\big[\ell_\theta(x,y)\big].
\end{equation}

We can decompose $`P(x,y)=P(x)P(y\mid x)`$, which gives a good interpretation.

In general, $`\hat{\theta}_0 \neq \theta^*`$ because the underlying measure in the expectation changes from $`P_0`$ to $`P^*`$. This mismatch manifests in several common failure modes:

(1) Covariate/domain shift. The input marginal differs, i.e., $`P_0(x)\neq P^*(x)`$, so regions that are frequent under $`P^*`$ may be rarely sampled under $`P_0`$. When the support overlaps, the target risk admits an importance-weighted form

\begin{equation}
\label{eq:iw}
\mathbb{E}_{(x,y)\sim P^*}\big[\ell_\theta(x,y)\big]
=
\mathbb{E}_{(x,y)\sim P_0}\!\left[\frac{P^*(x,y)}{P_0(x,y)}\,\ell_\theta(x,y)\right],
\end{equation}

which highlights that large density ratios $`\frac{P^*(x,y)}{P_0(x,y)}`$ correspond to regions that are under-covered by $`P_0`$ but can dominate the target risk.

(2) Support mismatch. A more severe problem occurs when there exists a region $`\mathcal{A}`$ such that

\begin{equation}
\label{eq:support_mismatch}
P^*(\mathcal{A})>0,\qquad P_0(\mathcal{A})=0,
\end{equation}

meaning that the target distribution assigns non-zero mass to inputs never seen during stage-1 training. In this case, ERM on $`P_0`$ cannot provide guarantees on $`\mathcal{A}`$. This would cause a extremely large density ratio.

(3) Underweighting high-loss regions. Even if $`P_0`$ and $`P^*`$ share support, uniform sampling under $`P_0`$ may assign very small probability mass to difficult, high-loss examples. As a result, a large fraction of gradient updates is spent on “easy” regions, while errors concentrated in a small but critical subset (often over-represented in target benchmarks) remain insufficiently optimized.

(4) Conditional shift in outputs. Finally, the conditional distribution can differ, i.e., $`P_0(y\mid x)\neq P^*(y\mid x)`$, which in LLM post-training often appears as differences in solution style, chain-of-thought structure, or answer formatting. Such mismatches can lead the model to learn a behavior prior that is misaligned with the target tasks.

How failure-driven retrieval-synthesis mitigates distribution mismatch.

Our second-stage data construction explicitly targets these issues by inducing a new training distribution $`P_{\mathrm{syn}}`$ from model failures observed on a target set $`Q`$ (ideally, $`Q\approx P^*`$). Failure-driven resampling shifts training focus toward regions where the current model performs poorly under the target distribution, mitigating covariate shift and the underweighting of high-loss regions. Moreover, embedding-based retrieval defines a kernel

\begin{equation}
\label{eq:retrieval_kernel_discuss}
K(d\mid x)\propto \exp\!\big(\langle \varphi(x),\varphi(d)\rangle/\tau\big),
\end{equation}

which maps failure queries to relevant external documents. By synthesizing new training instances conditioned on the retrieved documents, we effectively expand the support of the training data, addressing support mismatch by assigning non-zero probability to regions that were absent in $`P_0`$. Finally, the synthesis prompts and verification filters allow us to better align output formats and reasoning traces with target requirements, partially reducing conditional shift.

Failure-driven Post-training Provides Better Gradient Estimation

\begin{equation}
\label{eq:grad_sim}
\langle \nabla {\mathcal{L}}^*(\theta), \nabla{\mathcal{L}}_1(\theta)\rangle
\ge
\langle \nabla {\mathcal{L}}^*(\theta), \nabla {\mathcal{L}}_0(\theta)\rangle,
\end{equation}

\begin{equation}
{\mathcal{L}}^*(\theta-\eta \nabla {\mathcal{L}}_1(\theta)) \le {\mathcal{L}}^*(\theta-\eta \nabla {\mathcal{L}}_0(\theta)).
\end{equation}
```*

</div>

</div>

**Proof**: We first
prove <a href="#eq:grad_sim" data-reference-type="ref+label"
data-reference="eq:grad_sim">[eq:grad_sim]</a> as a lemma:

<div class="examplebox">

<div id="lemma:kernel_conditions" class="lemma">

**Lemma 3** (Conditions for Failure-Driven Gradient Alignment). *The
failure-driven gradient $`\nabla {\mathcal{L}}_1`$ achieves better
alignment with the target gradient $`\nabla {\mathcal{L}}^*`$ than the
vanilla gradient $`\nabla {\mathcal{L}}_0`$, i.e.,
``` math
\begin{equation}
    \langle \nabla {\mathcal{L}}^*(\theta), \nabla{\mathcal{L}}_1(\theta)\rangle \ge \langle \nabla {\mathcal{L}}^*(\theta), \nabla {\mathcal{L}}_0(\theta)\rangle,
\end{equation}

We partition the data space based on the performance of the current model $`\theta`$:

• Success Region ($`\mathcal{S}`$): $`\mathcal{S} = \{x \mid w_{\theta}(x) = 0\}`$. The model predicts correctly and the loss is negligible.

• Failure Region ($`\mathcal{F}`$): $`\mathcal{F} = \{x \mid w_{\theta}(x) = 1\}`$. The model fails; loss is high.

Let $`y^*(x)`$ denote the ground truth label (from $`P^*`$) for input $`x`$. We assume that in the success region $`\mathcal{S}`$, the model is well-converged. The expected gradient is dominated by zero-mean noise:

\begin{equation}
    \mathbb{E}_{x \sim P(\cdot|\mathcal{S})} [\nabla \ell_\theta(x, y^*(x))] \approx \mathbf{0}.
\end{equation}

Consequently, the target gradient is determined by the failure region. Let $`\mathbf{v}^*`$ be the average gradient direction required to fix the failures:

\begin{equation}
    \nabla {\mathcal{L}}^*(\theta) \approx P^*(\mathcal{F}) \cdot \underbrace{\mathbb{E}_{x \sim P^*(\cdot|\mathcal{F})} [\nabla \ell_\theta(x, y^*(x))]}_{\triangleq \mathbf{v}^*}.
\end{equation}

Let the synthetic gradient be $`\mathbf{v}_{\mathrm{syn}} = \mathbb{E}_{P_{\mathrm{syn}}}[\ell_\theta(x,y)]`$, and assume a similarity factor $`\alpha > 0`$:

\begin{equation}
        \label{eq:kernel_validity}
        \langle \mathbf{v}_{\mathrm{syn}}, \mathbf{v}^* \rangle \ge \alpha \|\mathbf{v}^*\|^2.
\end{equation}

Then we are able to compare the alignment of the vanilla gradient $`\nabla {\mathcal{L}}_0`$ and the failure-driven gradient $`\nabla {\mathcal{L}}_1`$ with the target gradient $`\nabla {\mathcal{L}}^*`$. The original training distribution $`P_0`$ is dominated by easy samples. Let $`\varepsilon= P_0(\mathcal{F})`$ be the proportion of failure cases in the original dataset, we have

\begin{equation}
 \nabla {\mathcal{L}}_0(\theta) = (1-\varepsilon)\mathbb{E}_{{\mathcal{S}}}[\ell_\theta] + \varepsilon\mathbb{E}_{\mathcal{F}}[\ell_\theta]\approx \mathbf{0} + \varepsilon\mathbf{v}^*.
\end{equation}

The inner product with the target gradient is:

\begin{equation}
\label{eq:align_g0}
    \langle \nabla {\mathcal{L}}^*,  \nabla {\mathcal{L}}_0 \rangle \approx \langle P^*(\mathcal{F})\mathbf{v}^*, \varepsilon\mathbf{v}^* \rangle = \varepsilon P^*(\mathcal{F}) \|\mathbf{v}^*\|^2.
\end{equation}

Then, recall $`P_1 = \lambda P_0 + (1-\lambda) P_{\mathrm{syn}}`$. The synthetic distribution $`P_{\mathrm{syn}}`$ concentrates mass on $`\mathcal{F}`$ and generates gradients $`\mathbf{v}_{\mathrm{syn}}`$.

\begin{equation}
    \nabla {\mathcal{L}}_1(\theta) = \lambda \nabla{\mathcal{L}}_0(\theta) + (1-\lambda) \mathbf{v}_{\mathrm{syn}}.
\end{equation}

The inner product with the target gradient is:

\begin{equation}
    \langle \nabla {\mathcal{L}}^*, \nabla {\mathcal{L}}_1 \rangle = \lambda \langle \nabla {\mathcal{L}}^*, \nabla{\mathcal{L}}_0 \rangle + (1-\lambda) \langle \nabla {\mathcal{L}}^*, \mathbf{v}_{\mathrm{syn}} \rangle.
\end{equation}

Substituting $`\nabla {\mathcal{L}}^* \approx P^*(\mathcal{F})\mathbf{v}^*`$:

\begin{equation}
\label{eq:align_g1}
    \langle \nabla {\mathcal{L}}^*, \mathbf{v}_{\mathrm{syn}} \rangle \approx P^*(\mathcal{F}) \langle \mathbf{v}^*, \mathbf{v}_{\mathrm{syn}} \rangle \ge \alpha P^*(\mathcal{F}) \|\mathbf{v}^*\|^2.
\end{equation}

From Eq. equation [eq:align_g0] and Eq. equation [eq:align_g1], it is realistic to assume $`\alpha \ge \varepsilon`$, then

\begin{equation}
    \alpha P^*(\mathcal{F}) \|\mathbf{v}^*\|^2 \ge \varepsilon P^*(\mathcal{F}) \|\mathbf{v}^*\|^2
\end{equation}

which implies that

\begin{equation}
\langle \nabla {\mathcal{L}}^*(\theta), \nabla{\mathcal{L}}_1(\theta)\rangle
\ge
\langle \nabla {\mathcal{L}}^*(\theta), \nabla {\mathcal{L}}_0(\theta)\rangle,
\end{equation}

Remark. The inequality holds when $`\alpha \ge \varepsilon\approx P_0(\mathcal{F})`$, where $`\alpha`$ is the similarity factor between synthetic gradient and true gradient (typically high approximating 1), and $`P_0(\mathcal{F})`$ is the error rate on the training set (typically very low (e.g., $`<0.1`$) as the model fits the training data). $`\alpha`$ is determined by cetrain properties of $`Q`$, $`K`$ and $`G`$, which we discussed as,

1. $`Q`$ has to approximate $`P^*`$. When $`Q \approx P^*`$, the evaluation distribution $`Q`$ effectively covers the target failure region, i.e., $`Q_{\theta}(\mathcal{F}) \approx 1`$.

2. $`K`$ has to correctly retrieve the relevant documents. So the retrieved $`z`$ are close to $`x`$ around the failure region.

3. $`G(x,y\mid d)`$ jointly need to be close to $`P^*(x,y)`$ and $`G(y'|d,x)`$ produces synthetic labels $`y'`$ that provide a descent direction aligned with the ground truth, i.e. $`y^\prime (x) \approx y^*(x)`$. If retrieval is irrelevant or synthesis hallucinates (invalid $`K, G`$), then $`\alpha \le 0`$ in [eq:kernel_validity], and the method fails.

\begin{equation}
\|\nabla {\mathcal{L}}^*(\theta)-\nabla {\mathcal{L}}^*(\theta')\|\le L\|\theta-\theta'\|.
\end{equation}

\begin{equation}
\theta_i^+ = \theta - \eta g_i,\quad i\in\{0,1\}.
\end{equation}

\begin{equation}
\langle \nabla {\mathcal{L}}^*(\theta), g_0\rangle \le\langle \nabla {\mathcal{L}}^*(\theta), g_1\rangle
 \Longrightarrow 
{\mathcal{L}}^*(\theta_1^+) \leqslant{\mathcal{L}}^*(\theta_0^+).
\end{equation}

Proof. Consider a single gradient-based update

\begin{equation}
\label{eq:gd_update}
\theta^+ = \theta - \eta g,
\end{equation}

where $`g`$ is the update direction. Using a first-order Taylor approximation of $`L^*`$ around $`\theta`$ yields

\begin{equation}
\label{eq:taylor_first_order}
{\mathcal{L}}^*(\theta^+) \approx {\mathcal{L}}^*(\theta) + \left\langle \nabla {\mathcal{L}}^*(\theta), \theta^+-\theta \right\rangle
= {\mathcal{L}}^*(\theta) - \eta \left\langle \nabla {\mathcal{L}}^*(\theta), g \right\rangle.
\end{equation}

Eq. equation [eq:taylor_first_order] shows that, for a fixed step size $`\eta`$, the target risk decreases more in one step when the inner product $`\langle \nabla L^*(\theta), g\rangle`$ is larger.

Assume that $`{\mathcal{L}}^*`$ is $`L`$-smooth (i.e., its gradient is $`L`$-Lipschitz). Then the descent lemma gives:

\begin{equation}
\label{eq:descent_lemma}
{\mathcal{L}}^*(\theta^+) \le
{\mathcal{L}}^*(\theta)
- \eta \left\langle \nabla {\mathcal{L}}^*(\theta), g \right\rangle
+ \frac{L\eta^2}{2}\|g\|^2.
\end{equation}

Therefore, for the same step size $`\eta`$, if $`\langle \nabla {\mathcal{L}}^*(\theta), g_0\rangle \leqslant\langle \nabla {\mathcal{L}}^*(\theta), g_1\rangle`$ makes the update reduce the target risk more effectively, i.e.,

\begin{equation}
{\mathcal{L}}^*(\theta_1^+) \leqslant{\mathcal{L}}^*(\theta_0^+).
\end{equation}

For stage-2 SFT we typically take

\begin{equation}
\label{eq:g_is_grad_Lsyn}
g_0 = \nabla {\mathcal{L}}_{0}(\theta),\quad g_1 = \nabla {\mathcal{L}}_{1}(\theta).
\end{equation}

Substituting ends the proof.

Extension to RL: Theory

Our failure-driven post-training framework can be naturally extends to reinforcement learning with verifiable rewards (RLVR). We first show that RL (with KL regularization) can be viewed as distribution matching towards a reward-tilted target distribution, and that failure-driven RL further improves the match to the gold distribution $`P^*`$ by correcting the mismatch.

RL with KL regularization induces a reward-tilted target distribution.

Let $`\pi_\theta(y\mid x)`$ denote the current policy and $`\pi_0(y\mid x)`$ be a reference policy (the stage-1 SFT model), which serves as a strong proposal distribution with broad support. Consider the following KL-regularized RL objective :

\begin{equation}
\max_{\pi(\cdot\mid x)}
\mathbb{E}_{y\sim \pi(\cdot\mid x)}\big[ A(x,y) \big]
-\frac{1}{\beta}\mathrm{KL}\!\left(\pi(\cdot\mid x)\,\|\,\pi_0(\cdot\mid x)\right),
\label{eq:kl_rl_objective}
\end{equation}

where $`A(x,y)`$ is the advantage of some verifiable reward (e.g., correctness/format/length), and $`\beta>0`$ controls the strength of the KL penalty. For any fixed $`x`$, the maximizer of [eq:kl_rl_objective] admits a closed form:

\begin{equation}
\pi^*_\beta(y\mid x)
=
\frac{\pi_0(y\mid x)\exp\big(\beta A(x,y)\big)}
{\sum_{y'}\pi_0(y'\mid x)\exp\big(\beta A(x,y')\big)}
\propto
\pi_0(y\mid x)\exp\!\big(\beta A(x,y)\big).
\label{eq:reward_tilted_optimal_policy}
\end{equation}

[eq:reward_tilted_optimal_policy] shows that KL-regularized RL performs an exponential tilting of the proposal $`\pi_0`$ by the advantage, thereby inducing an implicit target conditional distribution.

Moreover,  [eq:kl_rl_objective] is equivalent to a KL-based distribution matching problem:

\begin{equation}
\pi^*_\beta(\cdot\mid x)
=
\arg\min_{\pi(\cdot\mid x)}
\mathrm{KL}\!\left(\pi(\cdot\mid x)\,\|\,\pi^*_\beta(\cdot\mid x)\right).
\label{eq:kl_matching_form}
\end{equation}

Therefore, a second-stage RL procedure can be interpreted as pushing $`\pi_\theta(\cdot\mid x)`$ towards $`\pi^*_\beta(\cdot\mid x)`$, i.e., matching a target distribution $`P^*(y\mid x)`$.

Failure-driven RL improves the fit to $`P^*`$ by correcting the prompt marginal.

In practice, the overall goal is to reduce the target risk under the unknown gold distribution $`P^*(x,y)`$. Even if RL improves the conditional distribution $`P(y\mid x)`$, a mismatch on the prompt marginal can still lead to suboptimal optimization signal. Concretely, standard RL typically samples prompts from a surrogate prompt distribution (e.g., the stage-1 prompt marginal $`P_0(x)`$), while the true target prompt marginal under $`P^*`$ may emphasize different regions (especially hard STEM problems).

Our failure-driven data construction defines a new prompt distribution

\begin{equation}
P_1(x) \;=\; \lambda P_0(x) + (1-\lambda)P_{\mathrm{syn}}(x),
\label{eq:p1_prompt_marginal}
\end{equation}

As discussed previously, $`P_1(x)`$ better mimic the real distribution $`P^*(x)`$, thus provides a better estimation to the target distribution. As a result, RL updates are computed using samples that are more representative of the hard regions that dominate the target risk, yielding a better approximation to the ideal distribution-matching objective. Under $`P_1(x)`$, the KL-regularized RL objective becomes

\begin{equation}
\max_\theta
\;\;
\mathbb{E}_{x\sim P_1(x),\,y\sim \pi_\theta(\cdot\mid x)}\!\big[ A(x,y) \big]
\;-\;
\frac{1}{\beta}\,\mathbb{E}_{x\sim P_1(x)}\!\left[\mathrm{KL}\!\left(\pi_\theta(\cdot\mid x)\,\|\,\pi_0(\cdot\mid x)\right)\right].
\label{eq:rl_under_p1}
\end{equation}

Thus, analogous to [eq:reward_tilted_optimal_policy],  [eq:rl_under_p1] corresponds to matching the joint distribution

\begin{equation}
P^{*}_{\beta,1}(x,y)
\propto P_1(x)P_0(y\mid x)\exp\big(\beta A(x,y)\big),
\label{eq:joint_target_under_p1}
\end{equation}

where $`P_1(x)`$ is the surrogate prompt distribution and $`P^\prime(y\mid x)\propto P_0(y\mid x)\exp\big(\beta A(x,y)\big)`$ is the surrogate Response distribution. Overall $`P^{*}_{\beta,1}(x,y)`$ provides better estimated risk over the gold distribution $`P^*(x,y)`$ when $`P_1(x)`$ is a better surrogate of the target prompt marginal and $`P^\prime(y\mid x)`$ is a good Response distribution. . The proof follows almost the same as 9.2, thus we omit it here.