MixGRPO: Unlocking Flow-based GRPO Efficiency with Mixed ODE-SDE

Although GRPO substantially enhances flow matching models in human preference alignment of image generation, methods such as FlowGRPO and DanceGRPO still exhibit inefficiency due to the necessity of sampling and optimizing over all denoising steps specified by the Markov Decision Process (MDP). In this paper, we propose $\textbf{MixGRPO}$, a novel framework that leverages the flexibility of mixed sampling strategies through the integration of stochastic differential equations (SDE) and ordinary differential equations (ODE). This streamlines the optimization process within the MDP to improve efficiency and boost performance. Specifically, MixGRPO introduces a sliding window mechanism, using SDE sampling and GRPO-guided optimization only within the window, while applying ODE sampling outside. This design confines sampling randomness to the time-steps within the window, thereby reducing the optimization overhead, and allowing for more focused gradient updates to accelerate convergence. Additionally, as time-steps beyond the sliding window are not involved in optimization, higher-order solvers are supported for faster sampling. So we present a faster variant, termed $\textbf{MixGRPO-Flash}$, which further improves training efficiency while achieving comparable performance. MixGRPO exhibits substantial gains across multiple dimensions of human preference alignment, outperforming DanceGRPO in both effectiveness and efficiency, with nearly 50% lower training time. Notably, MixGRPO-Flash further reduces training time by 71%.

💡 Research Summary

MixGRPO addresses the inefficiency of existing flow‑matching reinforcement‑learning‑from‑human‑feedback (RLHF) methods such as Flow‑GRPO and DanceGRPO, which require full‑trajectory stochastic differential equation (SDE) sampling and policy optimization over every denoising step. The authors propose a mixed ODE‑SDE framework that confines stochastic sampling to a sliding window of timesteps while using deterministic ordinary differential equation (ODE) sampling elsewhere. By restricting GRPO optimization to the window, the effective horizon of the Markov decision process (MDP) is shortened, dramatically reducing the number of function evaluations (NFE) and focusing gradient updates on the most impactful steps.

The sliding window W(l) = {t_l,…,t_l+w‑1} moves from low‑signal‑to‑high‑signal regions as training progresses, implementing a curriculum that first optimizes high‑impact, high‑noise steps (global structure) and later refines low‑noise steps (fine details). Within the window, SDE updates follow the Euler‑Maruyama scheme:
x_{t+Δt}=x_t+μ_θ(x_t,t)Δt+σ_t√Δt ε,
where μ_θ combines the predicted velocity field v_θ and the score function s_t. Outside the window, deterministic ODE updates use x_{t+Δt}=x_t+v_θ(x_t,t)Δt.

The GRPO objective is applied only to the windowed timesteps:
J_{MixGRPO}(θ)=E_{c, {x_i^T}∼π_{θold}}