$V_0$: A Generalist Value Model for Any Policy at State Zero

Policy gradient methods rely on a baseline to measure the relative advantage of an action, ensuring the model reinforces behaviors that outperform its current average capability. In the training of Large Language Models (LLMs) using Actor-Critic methods (e.g., PPO), this baseline is typically estimated by a Value Model (Critic) often as large as the policy model itself. However, as the policy continuously evolves, the value model requires expensive, synchronous incremental training to accurately track the shifting capabilities of the policy. To avoid this overhead, Group Relative Policy Optimization (GRPO) eliminates the coupled value model by using the average reward of a group of rollouts as the baseline; yet, this approach necessitates extensive sampling to maintain estimation stability. In this paper, we propose $V_0$, a Generalist Value Model capable of estimating the expected performance of any model on unseen prompts without requiring parameter updates. We reframe value estimation by treating the policy’s dynamic capability as an explicit context input; specifically, we leverage a history of instruction-performance pairs to dynamically profile the model, departing from the traditional paradigm that relies on parameter fitting to perceive capability shifts. Focusing on value estimation at State Zero (i.e., the initial prompt, hence $V_0$), our model serves as a critical resource scheduler. During GRPO training, $V_0$ predicts success rates prior to rollout, allowing for efficient sampling budget allocation; during deployment, it functions as a router, dispatching instructions to the most cost-effective and suitable model. Empirical results demonstrate that $V_0$ significantly outperforms heuristic budget allocation and achieves a Pareto-optimal trade-off between performance and cost in LLM routing tasks.

💡 Research Summary

The paper tackles a fundamental inefficiency in reinforcement‑learning‑from‑human‑feedback (RLHF) pipelines for large language models (LLMs): the tight coupling between the policy (actor) and its value function (critic). In traditional actor‑critic methods such as PPO, the critic V ϕ(x) must be continuously retrained to track the non‑stationary policy π θ, which incurs heavy compute and memory overhead. Group Relative Policy Optimization (GRPO) removes the critic entirely by using the mean reward of a batch of rollouts as a baseline, but this shifts the burden to massive Monte‑Carlo sampling, especially problematic when rewards are sparse or collapse.

The authors propose V₀, a “Generalist Value Model” that estimates the expected performance of any LLM policy on a new prompt without any parameter updates. The key insight is to treat the policy’s capability as an explicit context Cπ = {(xᵢ, rᵢ)}ₙ consisting of historical instruction‑performance pairs, rather than an implicit latent variable embedded in the critic’s weights. Value estimation then becomes a conditional prediction problem V₀(x, Cπ) ≈ P(r = 1 | x, Cπ), which can be solved in a single forward pass.

Architecturally V₀ comprises three components:

1. Semantic‑Perception Backbone – a pre‑trained encoder maps both the target prompt x and each context prompt xᵢ into high‑dimensional embeddings h ∈ ℝ^{d_embed}. This captures semantic meaning, domain attributes, and difficulty cues.

2. Residual Query Adapter – because the downstream inference head (TabPFN) expects structured tabular features, the adapter projects the entangled embeddings into K fixed‑dimensional channels. It does so by combining a set of learnable static queries Q_static with dynamic offsets ΔQ = G(h) generated from the embedding, then applying multi‑head attention (MHA) to obtain structured features z ∈ ℝ^{K × d_embed}. This “semantic prism” aligns the representation with the Bayesian inference requirements of TabPFN.

3. Probabilistic In‑Context Head (TabPFN) – TabPFN treats the transformed context pairs {(zᵢ, rᵢ)} as observations and, in one forward pass, approximates the posterior predictive distribution P(r | z, {(zᵢ, rᵢ)}). The output is a success probability for the target prompt.

Training uses a composite loss to avoid a shortcut bias identified via mutual‑information analysis. Pure cross‑entropy encourages the model to rely solely on I(Y; C), i.e., the prior success rate implied by the context, ignoring the interaction I(Y; X | C). To counter this, the authors add a Bradley‑Terry pairwise ranking loss on logits from prompts within the same context, enforcing that the model distinguishes relative difficulty. The combination of soft cross‑entropy (for calibrated absolute probabilities) and ranking loss (for contextual reasoning) yields a model that genuinely leverages both the prompt and the capability history.

Empirical evaluation covers two scenarios:

• GRPO Training Budget Allocation – V₀ predicts success probabilities before rollouts, allowing the trainer to allocate more samples to hard prompts and fewer to easy ones. Compared with vanilla GRPO, V₀ reduces the total number of rollouts by over 30 % while achieving smoother reward curves and higher final performance.

• Inference‑Time Model Routing – In a multi‑model deployment (e.g., 7B, 13B, 34B LLMs), V₀ acts as a router that selects the cheapest model capable of handling a given instruction. Experiments show that V₀’s routing achieves a Pareto‑optimal trade‑off between cost and success rate, outperforming heuristic baselines and embedding‑distance routers by 2–3 % absolute success at the same compute budget.

In summary, V₀ reframes value estimation from a parameter‑fitting problem to an in‑context inference problem, decoupling the critic from the evolving policy. By leveraging a hybrid semantic‑to‑structured architecture and a carefully designed loss that mitigates shortcut learning, V₀ delivers both training‑time efficiency (less sampling, no critic updates) and deployment‑time efficiency (cost‑aware routing). The work opens avenues for extending context‑based value estimation to multi‑step horizons, non‑binary rewards, and larger, more heterogeneous model fleets.