Reinforcement Learning with Backtracking Feedback

Addressing the critical need for robust safety in Large Language Models (LLMs), particularly against adversarial attacks and in-distribution errors, we introduce Reinforcement Learning with Backtracking Feedback (RLBF). This framework advances upon prior methods, such as BSAFE, by primarily leveraging a Reinforcement Learning (RL) stage where models learn to dynamically correct their own generation errors. Through RL with critic feedback on the model’s live outputs, LLMs are trained to identify and recover from their actual, emergent safety violations by emitting an efficient “backtrack by x tokens” signal, then continuing generation autoregressively. This RL process is crucial for instilling resilience against sophisticated adversarial strategies, including middle filling, Greedy Coordinate Gradient (GCG) attacks, and decoding parameter manipulations. To further support the acquisition of this backtracking capability, we also propose an enhanced Supervised Fine-Tuning (SFT) data generation strategy (BSAFE+). This method improves upon previous data creation techniques by injecting violations into coherent, originally safe text, providing more effective initial training for the backtracking mechanism. Comprehensive empirical evaluations demonstrate that RLBF significantly reduces attack success rates across diverse benchmarks and model scales, achieving superior safety outcomes while critically preserving foundational model utility.

💡 Research Summary

The paper introduces Reinforcement Learning with Backtracking Feedback (RLBF), a novel framework designed to improve the safety of large language models (LLMs) against sophisticated adversarial attacks. RLBF builds on earlier work such as BSAFE but adds two major components: (1) an enhanced supervised fine‑tuning (SFT) data generation method called BSAFE+ and (2) a reinforcement‑learning (RL) stage that uses a dedicated safety critic to provide real‑time feedback during generation.

In the BSAFE+ pipeline, high‑quality safe responses are first generated by a capable base model. A safety‑violating segment is then programmatically inserted at a context‑ually coherent location within the safe answer. Because the original safe answer is known, the exact number of tokens that must be removed (the backtrack length X) is precisely defined. Training examples are constructed so that the model receives the prompt plus the safe prefix, and is required to output a category token (e.g.,