Answer First, Reason Later: Aligning Search Relevance via Mode-Balanced Reinforcement Learning

Building a search relevance model that achieves both low latency and high performance is a long-standing challenge in the search industry. To satisfy the millisecond-level response requirements of online systems while retaining the interpretable reasoning traces of Large Language Models (LLMs), we propose a novel \textbf{Answer-First, Reason Later (AFRL)} paradigm. This paradigm requires the model to output the definitive relevance score in the very first token, followed by a structured logical explanation. Inspired by the success of reasoning models, we adopt a “Supervised Fine-Tuning (SFT) + Reinforcement Learning (RL)” pipeline to achieve AFRL. However, directly applying existing RL training often leads to \textbf{mode collapse} in the search relevance task, where the model forgets complex long-tail rules in pursuit of high rewards. From an information theory perspective: RL inherently minimizes the \textbf{Reverse KL divergence}, which tends to seek probability peaks (mode-seeking) and is prone to “reward hacking.” On the other hand, SFT minimizes the \textbf{Forward KL divergence}, forcing the model to cover the data distribution (mode-covering) and effectively anchoring expert rules. Based on this insight, we propose a \textbf{Mode-Balanced Optimization} strategy, incorporating an SFT auxiliary loss into Stepwise-GRPO training to balance these two properties. Furthermore, we construct an automated instruction evolution system and a multi-stage curriculum to ensure expert-level data quality. Extensive experiments demonstrate that our 32B teacher model achieves state-of-the-art performance. Moreover, the AFRL architecture enables efficient knowledge distillation, successfully transferring expert-level logic to a 0.6B model, thereby reconciling reasoning depth with deployment latency.

💡 Research Summary

The paper tackles the long‑standing tension in search relevance systems between ultra‑low latency, high predictive performance, and interpretability. To meet millisecond‑level response requirements while preserving the transparent reasoning traces of large language models (LLMs), the authors introduce the Answer‑First, Reason‑Later (AFRL) paradigm. In AFRL the model must emit a definitive relevance score as the very first token, followed by a structured logical chain‑of‑thought (CoT) that explains the decision. This design eliminates the “time‑to‑first‑token” (TFT) latency that plagues conventional CoT approaches, because the first token already contains the actionable ranking signal. The subsequent reasoning trace serves as an audit trail for engineers and as a rich source of feedback during reinforcement learning (RL).

Applying RL directly to this task, however, leads to a phenomenon the authors call “mode collapse.” Standard RL optimizes the reverse Kullback‑Leibler (KL) divergence, a mode‑seeking objective that concentrates probability mass on high‑reward peaks. In a rule‑heavy relevance setting this encourages the model to discover simple shortcuts (e.g., keyword matching) and to forget complex, long‑tail expert rules. In contrast, supervised fine‑tuning (SFT) minimizes the forward KL divergence, a mode‑covering objective that forces the model to match the full data distribution, thereby preserving rare but important rules.

To reconcile these opposing forces, the authors propose a Mode‑Balanced Optimization strategy. They embed an auxiliary SFT loss into a Stepwise Group‑Relative Policy Optimization (GRPO) RL algorithm, forming a hybrid loss:

L_total(θ) = γ·L_SFT(θ) + α·L_GRPO(θ)

where γ and α weight the forward‑KL (mode‑covering) and reverse‑KL (mode‑seeking) components respectively. This “distributional anchor” prevents the RL phase from discarding expert knowledge while still allowing the model to sharpen its decision boundaries around high‑reward regions.

Data quality and rule coverage are ensured through two complementary mechanisms. First, the Policy Induction & Automated Refinement (PIAR) system implements an Act‑Diagnose‑Evolve loop: the model runs on a validation set, an evaluator diagnoses rule violations, and a refiner rewrites the instruction prompts accordingly. This automated instruction evolution reduces manual labeling effort while continuously improving the granularity and weighting of expert rules. Second, a multi‑stage curriculum learning schedule gradually introduces harder samples. Early stages focus on easy, high‑coverage examples to solidify basic relevance judgments; later stages inject long‑tail, nuanced cases, with stepwise advantage weighting that amplifies reward signals at critical checkpoints (the first‑token decision, boxed intermediate conclusions, and final confirmation).

The reward function is deterministic and rule‑based rather than learned. It combines a logic‑gate indicator (ensuring all nine reasoning steps are present and self‑consistent), an implicit CoT verification term, and an ordinal relevance penalty that scales with distance from the ground‑truth score. This strict formulation guarantees that a correct final score obtained through faulty reasoning receives zero reward, thereby discouraging reward hacking.

Experiments are conducted on large‑scale industrial search datasets. A 32‑billion‑parameter “teacher” model trained with the AFRL pipeline achieves state‑of‑the‑art performance on standard relevance metrics (NDCG, MRR, etc.), outperforming prior generative ranking models such as RankGPT and RankLlama. Importantly, the first‑token relevance score alone already yields high accuracy, confirming the efficacy of the AFRL design.

To address deployment constraints, the authors perform knowledge distillation from the 32B teacher to a 0.6B “student” model. The distillation objective mirrors the AFRL structure, teaching the student both the immediate relevance score and the subsequent reasoning trace. The distilled model retains most of the teacher’s effectiveness while reducing latency by an order of magnitude, making it suitable for real‑time production environments.

In summary, the paper contributes four key innovations: (1) the AFRL paradigm that decouples latency from reasoning depth, (2) a Mode‑Balanced Optimization that jointly minimizes forward and reverse KL divergences to avoid mode collapse, (3) an automated instruction evolution (PIAR) and curriculum framework that efficiently internalize expert rules, and (4) a successful teacher‑student distillation pipeline that delivers industrial‑grade relevance performance within strict latency budgets. The approach promises broader applicability to any domain where fast, interpretable, and rule‑consistent decisions are required.