GMTS: Gradient Magnitude-based Token Selection Improves RLVR Training for LLM Reasoning
A paper introduces Gradient Magnitude-based Token Selection (GMTS) for RLVR training of LLMs. It finds that high-entropy tokens correlate with large gradient magnitude within an answer, but entropy alone fails to consistently reflect token importance across different answers. Training on the top 20% tokens ranked by GMTS consistently outperforms entropy-based token selection across three reasoning domains.
The paper proposes GMTS, a method that leverages the entropy-gradient connection to approximate gradient-magnitude rankings for token selection in RLVR training. It addresses the limitation of entropy-based selection by accounting for answer-level reward signal variations. Experimental results show consistent improvement over entropy-based selection across three reasoning domains.
GMTS uses gradient magnitude as a more reliable token importance measure than entropy, especially across answers with varying reward signals. The method approximates gradient-magnitude rankings via the entropy-gradient connection, enabling efficient token selection without full gradient computation. Training on top 20% tokens by GMTS yields better reasoning performance.
This research suggests that token-level training efficiency can be improved in RLVR pipelines, potentially reducing compute costs while enhancing reasoning capabilities. It may influence how AI labs design RL training data selection strategies for reasoning models.
Improved token selection can lower training costs and improve model reasoning performance, offering competitive advantages for AI companies developing reasoning-focused LLMs.
Further validation on diverse reasoning benchmarks and integration with existing RLVR frameworks could establish GMTS as a standard token selection technique. Open-source implementations may accelerate adoption.