Self-Generative Adversarial Fine-Tuning for Large Language Models

Fine-tuning large language models (LLMs) for alignment typically relies on supervised fine-tuning or reinforcement learning from human feedback, both limited by the cost and scarcity of high-quality annotations. Recent self-play and synthetic data approaches reduce this dependence but often rely on heuristic assumptions or ungrounded self-evaluation, which can cause bias accumulation and performance drift. In this paper, we propose Self-Generative Adversarial LLM (SGALM), a unified fine-tuning framework that formulates alignment as a generative adversarial game within a single LLM. SGALM jointly evolves generation and discrimination capabilities without external reward models. Theoretical and empirical results demonstrate that SGALM achieves state-of-the-art performance, serves as an effective alignment algorithm and a robust synthetic data engine.

💡 Research Summary

The paper introduces SGALM (Self‑Generative Adversarial LLM), a unified fine‑tuning framework that treats alignment of large language models as a generative‑adversarial game played entirely within a single model. Traditional alignment pipelines—Supervised Fine‑Tuning (SFT), Reinforcement Learning from Human Feedback (RLHF), Direct Preference Optimization (DPO)—require massive amounts of high‑quality human annotations, which are costly and scarce. Recent self‑play or synthetic‑data methods alleviate this dependence but typically rely on heuristic assumptions (e.g., all model‑generated text is inferior) or external reward models that can drift, leading to bias accumulation and performance degradation.

SGALM eliminates external reward models by letting the LLM serve simultaneously as generator G and discriminator D, sharing the same parameter set θ. Generation is performed via few‑shot in‑context learning (ICL): a random subset of real examples is placed in a prompt (Z_ctx) together with a generation instruction (“generate a new example following the pattern”). The model samples a synthetic example z′ from the conditional distribution pθ(z′ | G Prompt, Z_ctx). Diversity arises from random selection of context examples and stochastic token sampling.

Discrimination is achieved by prompting the same model with a binary question (“Is the example Real (human‑written) or Fake (LLM‑generated)? Answer with one word”). The model’s output probability for “Real” (p_realθ(z)) is taken as a continuous score D(z)∈(0,1). No additional classifier or scoring head is required, preserving the model’s general‑intelligence behavior.

Training follows the classic GAN minimax objective:

• Discriminator loss J(D) = −E_{z∼p_T}