OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignment

Reward modeling lies at the core of reinforcement learning from human feedback (RLHF), yet most existing reward models rely on scalar or pairwise judgments that fail to capture the multifaceted nature of human preferences. Recent studies have explored rubrics-as-rewards (RaR) that uses structured criteria to capture multiple dimensions of response quality. However, producing rubrics that are both reliable and scalable remains a key challenge. In this work, we introduce OpenRubrics, a diverse, large-scale collection of (prompt, rubric) pairs for training rubric-generation and rubric-based reward models. To elicit discriminative and comprehensive evaluation signals, we introduce Contrastive Rubric Generation (CRG), which derives both hard rules (explicit constraints) and principles (implicit qualities) by contrasting preferred and rejected responses. We further remove noisy rubrics via preserving preference-label consistency. Across multiple reward-modeling benchmarks, our rubric-based reward model, Rubric-RM, surpasses strong size-matched baselines by 8.4%. These gains transfer to policy models on instruction-following and biomedical benchmarks.

💡 Research Summary

This paper tackles a fundamental limitation of current reinforcement learning from human feedback (RLHF) pipelines: the reliance on scalar scores or simple pairwise preferences that cannot fully capture the nuanced, multi‑dimensional nature of human judgments. Building on the emerging Rubrics‑as‑Rewards (RaR) paradigm, the authors introduce OpenRubrics, a large‑scale, diverse collection of (prompt, rubric) pairs, and a novel Contrastive Rubric Generation (CRG) method that simultaneously produces two complementary rubric types—hard rules (explicit, objective constraints) and principles (higher‑level, implicit quality criteria).

CRG works by presenting an instruction‑tuned language model with a prompt together with a ranked list of candidate responses (derived from chosen vs. rejected outputs). The model is asked to generate a set of evaluation criteria that explain why higher‑ranked responses are preferred. By leveraging the contrast between good and bad answers, the generated rubrics become both discriminative and comprehensive. To ensure that the automatically created rubrics faithfully reflect human preferences, the authors apply a Preference‑label Consistency filter: each rubric is re‑used to predict preferences on all induced response pairs, and only those that achieve at least 50 % agreement with the original human labels are retained for training.

The OpenRubrics dataset aggregates six publicly available sources—UltraFeedback, Magpie, Skywork‑Preference, Synthetic‑IF, MegaScience, and Medical‑o1—covering general instruction following, reasoning, and domain‑specific tasks such as scientific and medical question answering. For each prompt, multiple candidate answers are generated using three strong open‑source LLMs (Qwen‑3, LLaMA‑3.1, Gemma‑3). This yields 35.7 k prompts with an average of 4–6 hard rules and principles per prompt, providing broad coverage of topics, lengths, and rubric structures.

Two models are then trained. First, a rubric‑generation model gθ is fine‑tuned to map prompts to high‑quality rubrics. Second, a rubric‑conditioned reward model Rubric‑RM (rϕ) takes a prompt, a pair of responses, and the associated rubric as input and predicts the binary preference. Experiments on eight standard reward‑modeling benchmarks (including TruthfulQA, MT‑Bench, and AlpacaEval) show that Rubric‑RM consistently outperforms size‑matched scalar or pairwise baselines by an average of 8.4 % in preference‑prediction accuracy.

When Rubric‑RM is used as the reward signal for PPO‑based policy training, notable gains are observed on downstream tasks. LLaMA‑13B and Alpaca‑7B models trained with Rubric‑RM achieve 2–5 percentage‑point improvements in instruction‑following accuracy and 4 percentage‑point gains on a medical diagnostic benchmark. Ablation studies confirm that the combination of hard rules and principles is crucial: hard rules curb overly verbose outputs, while principles encourage logical soundness and factual correctness.

The paper also discusses limitations. The quality of generated rubrics still depends on the underlying LLM’s knowledge, which may be insufficient for highly specialized domains. The Preference‑label Consistency threshold (τ = 0.5) is a hyper‑parameter that trades off between data quantity and noise removal. Future work is suggested in three directions: (1) incorporating expert validation to further refine rubrics, (2) extending CRG to handle multi‑user or multi‑objective preference settings, and (3) applying rubric‑based rewards to multimodal models.

In summary, OpenRubrics and the CRG framework provide a scalable, cost‑effective pipeline for synthesizing high‑quality, interpretable rubrics that enhance both the performance and transparency of reward models, representing a significant step forward for LLM alignment.