Muon · Jul 17, 2026
When Does Muon Help Agentic Reinforcement Learning?
An exploratory study on ALFWorld sparse-reward agentic reinforcement learning tasks, using the Qwen2.5-0.5B-Instruct model, compared the Muon optimizer with AdamW. Under Group-in-Group Policy Optimization (GiGPO), applying Muon only to hidden weight matrices improved final window validation success rate from 0.290 to 0.546 (+88%). The effect depends on advantage estimator and learning rate: at learning rate 3e-5, Muon improved GRPO success rate from 0.161 to 0.268; at learning rate 1e-5, GraphGPO Muon achieved 0.901, normalized validation AUC increased from 0.399 to 0.556, and reaching 0.5 and 0.75 success rates was 30 and 60 updates earlier, respectively.
What happened
The Muon optimizer shows potential in agentic reinforcement learning post-training, significantly improving success rates on ALFWorld tasks, with effects influenced by advantage estimator and learning rate.
Technical significance
The Muon optimizer may improve policy optimization stability in RL fine-tuning by enhancing hidden layer weight updates, but its effectiveness is highly dependent on the choice of advantage estimator and learning rate, suggesting that optimizer and RL algorithm components need joint tuning.
Industry impact
If Muon proves effective across more RL tasks, it could become a standard optimizer for RL post-training, lowering the barrier for agent development and accelerating the commercialization of embodied intelligence and autonomous agents.
What to watch
Multi-seed and cross-task validation are needed to confirm Muon's generalization; future research may focus on joint automatic tuning of optimizer-estimator-learning rate, and Muon's application in larger models and more complex environments.
Decision value
Improving RL training efficiency and success rates may reduce computational cost and time for developing agent systems, with potential commercial value in robotics, game AI, and autonomous agents.