Inverse Reinforcement Learning with Dynamic Reward Scaling for LLM Alignment

Alignment is vital for safely deploying large language models (LLMs). Existing techniques are either reward-based (training a reward model on preference pairs and optimizing with reinforcement learning) or reward-free (directly fine-tuning on ranked outputs). Recent research shows that well-tuned reward-based pipelines remain the most robust, and single-response demonstrations can outperform pairwise preference data. However, there still exist two key challenges: (1) imbalanced safety datasets that overrepresent common hazards while neglecting long-tail threats; and (2) static reward models that ignore task difficulty, limiting optimization efficiency and attainable gains. To address these limitations, we propose DR-IRL, which Dynamically adjusts Rewards through Inverse Reinforcement Learning. We first train category-specific reward models using a balanced safety dataset of seven harmful categories as demonstration via IRL. Then we enhance Group Relative Policy Optimization (GRPO) by introducing dynamic reward scaling: adjusting rewards by task difficulty, data-level hardness by text encoder cosine similarity, and model-level responsiveness by reward gaps. Extensive experiments across various benchmarks and LLMs demonstrate that DR-IRL outperforms all baseline methods in safety alignment while maintaining usefulness.

💡 Research Summary

The paper introduces DR‑IRL, a novel framework for aligning large language models (LLMs) that addresses two persistent problems in current alignment pipelines: (1) safety datasets that are heavily skewed toward common hazards and neglect long‑tail threats, and (2) static reward models that treat all training examples equally regardless of task difficulty.

First, the authors construct a balanced safety demonstration dataset covering seven harmful categories (e.g., violence, sexual content, misinformation) using a “Chain‑of‑Draft” (CoD) prompting strategy that lets an LLM generate refusal responses for each category. This approach replaces costly pairwise preference data with single‑response demonstrations, dramatically reducing annotation effort while preserving human value signals.

Next, they train category‑specific “shadow” reward models via Inverse Reinforcement Learning (IRL). Using a maximum‑likelihood IRL (ML‑IRL) formulation, they set up a bilevel optimization where the policy πθ maximizes expected reward minus a KL‑regularization term, and the reward function r(x, y; θ) is learned from the demonstration set. The resulting reward models capture nuanced safety judgments for each hazard type.

For the alignment phase, the authors augment Group Relative Policy Optimization (GRPO) with a dynamic reward‑scaling mechanism. Two hardness signals are computed for every training sample:

• Data hardness (αD) – measured by the cosine similarity between the reference safe response and the model‑generated response, using a pretrained text encoder. The similarity is transformed into a difficulty score via a sigmoid and normalized across the category.

• Model responsiveness (αM) – measured by the reward gap R = r(q, o) – r(q, e) produced by the shadow reward model, where o is the reference safe answer and e is the model’s current answer. Outliers are masked, the average gap is computed, and a sigmoid‑scaled responsiveness score is derived.

The final hardness coefficient α = αD·αM multiplies the advantage term in GRPO, ensuring that samples that are both semantically hard and still uncertain for the model receive higher learning weight, while trivial or already mastered cases receive less pressure. This multiplicative gating stabilizes training and focuses updates on long‑tail, high‑impact safety failures.

Experiments span multiple LLM architectures (LLaMA‑3, GPT‑Neo, Alpaca) and a suite of safety benchmarks (Harmless‑Eval, Jailbreak‑Bench, Red‑Team attacks) as well as utility benchmarks (MMLU, HumanEval, CodeGen). DR‑IRL consistently outperforms state‑of‑the‑art reward‑based methods (PPO‑HF, DPO) and reward‑free approaches, achieving 12‑18 percentage‑point gains in safety metrics while maintaining or slightly improving utility scores. Ablation studies confirm that both the IRL‑derived reward models and the dynamic scaling contribute to the performance boost.

The paper’s contributions are threefold: (1) a cost‑effective demonstration‑based IRL pipeline for category‑specific reward learning, (2) a dual‑signal dynamic reward‑scaling technique that integrates data difficulty and model confidence, and (3) extensive empirical validation showing that safety can be substantially improved without sacrificing usefulness.

Limitations include reliance on the quality of automatically generated demonstrations, potential over‑fitting of shadow reward models, and the need to tune hyperparameters governing hardness computation (e.g., masking threshold τ, percentile T). Future work may explore hybrid demonstration‑preference datasets, regularization strategies for reward models, and extensions to multimodal LLMs.