Headroom-Drift Replay: A Primitive for Principled Replay Control in GRPO
The paper introduces Headroom-Drift Replay, a group-level replay control primitive for GRPO that separates reuse into two decisions: Headroom ranks stored groups by remaining learning value, while Drift gates them by compatibility with the current policy. The fresh on-policy stream remains unchanged, and the method adds no auxiliary generation or training machinery. Across mathematical reasoning, multimodal reasoning, and Agentic Search benchmarks, this single intervention outperforms naive replay and matches or exceeds broader replay methods on Avg Mean@32. In Agentic Search, where environment interaction dominates cost, it delivers comparable quality.
RL-based post-training for reasoning models is increasingly bottlenecked by repeated fresh rollout generation, particularly in agentic settings where environment interaction dominates wall-clock cost. Replay can reduce this burden by reusing past trajectories, but existing methods typically embed it within larger training pipelines involving exploration, experience restructuring, or mixed-policy optimization. This makes replay's own contribution difficult to isolate. The paper asks a focused question: how far can principled replay selection alone go? It introduces Headroom-Drift Replay, a group-level replay control primitive for GRPO that separates reuse into two decisions. Headroom ranks stored groups by remaining learning value, while Drift gates them by compatibility with the current policy. The fresh on-policy stream remains unchanged, and the method adds no auxiliary generation or training machinery. Across mathematical reasoning, multimodal reasoning, and Agentic Search benchmarks, this single intervention outperforms naive replay and matches or exceeds broader replay methods on Avg Mean@32. In Agentic Search, where environment interaction dominates cost, it delivers comparable quality.
Headroom-Drift Replay isolates replay selection as a standalone primitive for GRPO, decoupling the value of stored trajectories (Headroom) from their compatibility with the current policy (Drift). This separation allows principled reuse without altering the on-policy stream or adding auxiliary machinery, suggesting that careful selection alone can capture much of the benefit of more complex replay pipelines.
The method directly targets the cost bottleneck of repeated fresh rollout generation in agentic RL, where environment interaction dominates wall-clock cost. By enabling effective replay with no additional generation or training overhead, it offers a practical path to reduce compute and time for post-training reasoning models, potentially lowering barriers for smaller teams and accelerating iteration cycles.
For organizations training reasoning models, especially in agentic settings, this approach can reduce the cost and time of RL post-training by reusing past trajectories without sacrificing quality. This may lower infrastructure expenses, speed up model development cycles, and make advanced reasoning model training more accessible to teams with limited compute budgets.
Observable next signals include follow-up work applying Headroom-Drift Replay to other RL algorithms beyond GRPO, empirical comparisons on additional agentic benchmarks, and adoption in open-source training frameworks. If the primitive proves robust, it may become a standard component in RL post-training pipelines, with further refinements to the Headroom and Drift metrics.