A First-Order Logic-Based Alternative to Reward Models in RLHF

Reinforcement Learning from Human Feedback (RLHF) plays a crucial role in aligning large language models (LLMs) with human values and preferences. However, the quality and stability of the trained reward model largely determine the final alignment performance. Existing approaches such as Proximal Policy Optimization (PPO) rely heavily on reward models to guide LLMs toward human-aligned behaviors. In this work, we propose a logic-similarity-based reward mechanism as an alternative to conventional reward modeling. Instead of relying on heuristic reward estimation, our method leverages formal logical consistency to steer model alignment with human preferences. Since real-world questions can be interpreted from multiple perspectives, to ensure that logic-based reinforcement learning does not cause model collapse, we introduce S-GRPO, a supervised variant of the GRPO framework. S-GRPO incorporates an additional supervised component and jointly optimizes the generation term, KL-divergence regularization, and label-based objective during training. Experimental results demonstrate that S-GRPO consistently outperforms standard supervised fine-tuning (SFT) in both performance and robustness. Furthermore, it extends existing preference-learning frameworks such as GRPO and DPO, offering a more flexible and task-adaptive approach to alignment training. Our code is available at https://github.com/ChunjinJiang/sgrpo.

💡 Research Summary

The paper addresses a central bottleneck in current Reinforcement Learning from Human Feedback (RLHF) pipelines: the reliance on external reward models to provide training signals for aligning large language models (LLMs) with human preferences. While recent methods such as Group Relative Policy Optimization (GRPO) have eliminated the need for a critic network, they still depend on a separate reward model, which adds computational overhead and can become a source of instability.

To overcome this, the authors introduce two complementary innovations. First, they propose Supervised‑GRPO (S‑GRPO), a supervised variant of GRPO that augments the original GRPO loss with a label‑based supervision term. The new loss function is:

L_SGRPO(θ) = E_{q,y∼P(Q)}