Alleviating Sparse Rewards by Modeling Step-Wise and Long-Term Sampling Effects in Flow-Based GRPO

Deploying GRPO on Flow Matching models has proven effective for text-to-image generation. However, existing paradigms typically propagate an outcome-based reward to all preceding denoising steps without distinguishing the local effect of each step. Moreover, current group-wise ranking mainly compares trajectories at matched timesteps and ignores within-trajectory dependencies, where certain early denoising actions can affect later states via delayed, implicit interactions. We propose TurningPoint-GRPO (TP-GRPO), a GRPO framework that alleviates step-wise reward sparsity and explicitly models long-term effects within the denoising trajectory. TP-GRPO makes two key innovations: (i) it replaces outcome-based rewards with step-level incremental rewards, providing a dense, step-aware learning signal that better isolates each denoising action’s “pure” effect, and (ii) it identifies turning points-steps that flip the local reward trend and make subsequent reward evolution consistent with the overall trajectory trend-and assigns these actions an aggregated long-term reward to capture their delayed impact. Turning points are detected solely via sign changes in incremental rewards, making TP-GRPO efficient and hyperparameter-free. Extensive experiments also demonstrate that TP-GRPO exploits reward signals more effectively and consistently improves generation. Demo code is available at https://github.com/YunzeTong/TurningPoint-GRPO.

💡 Research Summary

The paper addresses a fundamental limitation of existing GRPO‑based fine‑tuning for Flow Matching (FM) models used in text‑to‑image generation. Current methods such as Flow‑GRPO and DanceGRPO assign a single terminal reward—computed from the final clean image—to every intermediate denoising step. This “outcome‑based” reward leads to two problems: (1) reward sparsity, because the same scalar is used for all timesteps, preventing the model from learning the true contribution of each step; and (2) neglect of implicit cross‑step interactions, as the group‑wise ranking only compares trajectories at the same timestep, ignoring how early actions affect later states through the deterministic ODE completion.

To solve these issues, the authors propose TurningPoint‑GRPO (TP‑GRPO), which introduces two key innovations. First, they replace the terminal reward with an incremental step‑wise reward: for each SDE sampling step they compute the difference between the reward of the image obtained after the step and the reward before the step (using the same evaluation model). This dense signal directly reflects the “pure” gain of the individual denoising action, eliminating the sparsity of the original design.

Second, they identify turning points—steps where the local reward trend flips to become consistent with the overall trajectory trend. A turning point is detected purely by a sign change in the incremental reward (no magnitude threshold, no extra hyper‑parameters). For such steps they assign an aggregated long‑term reward that captures the delayed impact on all subsequent steps, effectively performing credit assignment over the whole trajectory.

Algorithmically, TP‑GRPO keeps the GRPO framework’s group advantage normalization but substitutes the per‑step advantage with either the incremental reward (for normal steps) or the aggregated long‑term reward (for turning points). This encourages the policy to favor actions that not only improve the immediate reward but also set the trajectory on a better long‑term path.

Experiments on several benchmarks (MS‑COCO, LAION‑Aesthetics, etc.) and evaluation metrics (FID, CLIP‑Score, human preference) show consistent improvements over baseline Flow‑GRPO. The gains are especially pronounced for prompts that induce high reward volatility, indicating that the method better handles complex, multi‑object scenes. Computational overhead is negligible because turning‑point detection only requires checking the sign of the incremental reward, and the approach is hyper‑parameter‑free, simplifying deployment.

In summary, TP‑GRPO simultaneously mitigates reward sparsity and introduces explicit modeling of delayed credit assignment in flow‑based generative models. By providing a dense, step‑aware learning signal and rewarding critical turning points, it yields more stable and higher‑quality image generation while preserving the simplicity and efficiency of the original GRPO pipeline. The code and reproducible experiments are publicly released.