Dialogue Model Optimization via Agent Game and Adaptive Tree-based GRPO

Open-ended dialogue agents aim to deliver engaging, personalized interactions by adapting to users’ traits, but existing methods face critical limitations: over-reliance on pre-collected user data, and short-horizon biases in reinforcement learning (RL) that neglect long-term dialogue value. To address these, we propose a novel long-horizon RL framework integrating online personalization with Adaptive Tree-based Group Relative Policy Optimization (AT-GRPO). Adopting a two-agent game paradigm, a user agent constructs dynamic environments via style mimicry (learning user-specific conversational traits) and active termination (predicting turn-level termination probabilities as immediate rewards), forming an iterative cycle that drives the dialogue agent to deepen interest exploration. AT-GRPO reinterprets dialogue trajectories as trees and introduces adaptive observation ranges. Unlike full tree expansion that incurs exponential overhead, it limits each node to aggregate rewards from a stage-aware range: larger ranges support early-stage topic exploration, while smaller ranges facilitate late-stage dialogue maintenance. This design reduces rollout budgets from exponential to polynomial in the dialogue length, while preserving long-term reward capture. Extensive experiments show our framework’s superior performance, sample efficiency, and robustness.

💡 Research Summary

The paper tackles two fundamental shortcomings of current open‑domain dialogue systems: (1) an over‑reliance on pre‑collected user data for personalization, and (2) short‑horizon bias in reinforcement‑learning (RL) approaches that neglect the long‑term value of conversations. To address both issues simultaneously, the authors propose a novel framework that combines an “agent game” architecture with an Adaptive Tree‑based Group Relative Policy Optimization (AT‑GRPO) algorithm.

Agent Game Architecture
The system consists of two interacting agents. The user agent simulates a real user by (a) learning conversational style through supervised fine‑tuning (SFT) on a small set of user‑specific dialogues (style mimicry) and (b) predicting a turn‑level termination probability at each step. The termination probability (p_i) is transformed into an immediate reward (r_i = 1 - p_i) for the dialogue agent, encouraging the latter to keep the conversation alive. Moreover, the user agent propagates (p_i) as an explicit feature in the dialogue context, allowing the dialogue agent to perceive the user’s willingness to continue. A dynamic threshold parameter (\alpha) linearly increases with RL training steps, making the user agent progressively stricter: early in training the user tolerates low‑quality responses, later it demands higher quality, thereby creating an adversarial training loop that pushes the dialogue agent toward continual improvement.

Adaptive Tree‑based GRPO (AT‑GRPO)
Traditional GRPO uses chain‑based rollouts, sampling a group of size (W) at each turn and continuing with a single selected node, resulting in a complexity of (O(W^L)) for a dialogue of length (L). While TreeRPO introduced bottom‑up weighted reward aggregation over a full tree, it suffers from exponential growth because every node must access all its descendants. AT‑GRPO mitigates this by introducing an adaptive observation range. For each node (n_{i,j}) the algorithm only aggregates rewards from child nodes that lie within a stage‑dependent window ((w, l)). In early dialogue stages the window is large, allowing the model to explore many possible topics and capture long‑term potential rewards. In later stages the window shrinks, focusing computation on maintaining and deepening already‑chosen topics. This adaptive truncation reduces the rollout budget from exponential to polynomial in (L) while preserving the essential bottom‑up reward signal that conveys long‑term value.

Reward Design and Robustness
The immediate reward (r_i = 1 - p_i) directly reflects the user agent’s termination likelihood. Because the termination probability is also fed back as a contextual feature, the dialogue agent can adjust its strategy based on the evolving user tolerance. The dynamic increase of (\alpha) prevents “reward hacking” behaviors such as meaningless repetitions or artificially prolonging the conversation without improving content quality.

Experimental Evaluation
The authors evaluate the framework on three datasets: LCCC (Chinese open‑domain chat), DailyDialog (English multi‑turn dialogues), and a custom game NPC dataset. Metrics include (1) interaction length, (2) coherence and logical consistency (both automatic metrics and human judgments), and (3) user satisfaction scores collected via surveys. Results show that AT‑GRPO‑trained agents achieve 15‑20 % longer dialogues and roughly 10 % higher coherence scores compared to strong baselines (PPO, standard GRPO) while requiring only about half the amount of training data to reach comparable performance. Notably, the system attains these gains after only 100 RL steps, demonstrating strong sample efficiency.

Contributions

1. Agent‑Game Framework for Online Personalization – eliminates the need for offline user histories by learning user traits on‑the‑fly through style mimicry and termination prediction.
2. AT‑GRPO Algorithm – introduces adaptive tree expansion and stage‑aware observation ranges to capture long‑term dialogue value without prohibitive computational cost.
3. Comprehensive Evaluation Protocol – validates the approach across diverse domains and metrics, establishing robustness and superiority over existing methods.

Future Directions
Potential extensions include incorporating multimodal user signals (voice tone, facial expressions), scaling the user agent to simulate multiple concurrent user personas, and deploying the system in real‑world applications to test online adaptation under live traffic. Overall, the paper presents a compelling solution to the twin challenges of data‑efficient personalization and long‑horizon optimization in open‑domain conversational AI.