SPO++: Stream-Aligned Policy Optimization for Asynchronous Agentic RL
SPO++ is a reinforcement learning method that improves upon Single-stream Policy Optimization (SPO) by standardizing terminal-outcome advantages under the action-token measure and organizing prompt evidence by the policy event that generated it. Across matched runs on ALFWorld at two model scales and on Math-TIR, SPO++ improves online learning efficiency over SPO. A paired ablation identifies action-token-measure normalization as the strongest tested component.
SPO++ addresses a mismatch in SPO where trajectory centering does not center the token-weighted quantity consumed by the actor. It fixes this by standardizing terminal-outcome advantages under the action-token measure and organizing prompt evidence by the generating policy event. Experiments on ALFWorld and Math-TIR show improved online learning efficiency over SPO.
The key technical contribution is the correction of advantage normalization: trajectory centering is replaced by standardization under the action-token measure, aligning the advantage distribution with the actor's token-weighted loss. Additionally, evidence is organized by the policy event that generated it rather than learner receipt order, which may reduce variance in asynchronous settings.
This work targets asynchronous agentic reinforcement learning, where long and variable tool-use trajectories make group-relative methods costly. By removing the need to wait for sibling rollouts, SPO++ could enable more efficient training of agentic models in real-world tool-use scenarios.
Improved online learning efficiency can reduce compute and time costs for training agentic models, potentially accelerating development cycles and lowering barriers for deploying RL-trained agents in commercial applications.
Future work may explore scaling SPO++ to larger models and more complex agentic environments, as well as further ablations on evidence organization strategies. The method's efficiency gains could influence adoption in production RL pipelines for tool-using agents.