Sequential Beats Joint: On the Interplay between On-Policy Distillation and RLVR
A paper on arXiv (cs.AI) shows that a two-stage scheme, OPD-then-RL, consistently outperforms pure OPD, pure RLVR, and joint baselines across logic and math reasoning benchmarks. The OPD validation score is identified as the key signal for when to switch to RL, and OPD is a better cold start for RL than SFT.
The paper investigates the combination of on-policy distillation (OPD) and reinforcement learning with verifiable rewards (RLVR) for post-training reasoning LLMs. It finds that a simple two-stage approach—OPD followed by RL—outperforms pure OPD, pure RLVR, and joint methods that fuse the two signals in a single step. The authors provide a systematic explanation based on pass@k behavior, learning dynamics, and parameter updates: OPD expands the student's coverage of teacher-supported solutions, and RL sharpens within that support, while joint optimization causes interference. Practical guidance includes using the OPD validation score to decide when to switch to RL and using OPD as a cold start instead of SFT.
The paper demonstrates that sequential training (OPD then RL) avoids interference between dense token-level distillation loss and sparse RL reward, leading to better reasoning performance. The OPD validation score serves as a reliable early stopping signal for transitioning to RL, and OPD initialization provides a more effective starting point than supervised fine-tuning.
This research suggests a practical recipe for improving reasoning LLMs: first apply on-policy distillation to broaden solution coverage, then apply RLVR to sharpen within that coverage. The finding that OPD is a better cold start than SFT could influence training pipelines for reasoning models.
The proposed OPD-then-RL method offers a simple and effective way to improve reasoning LLM performance, potentially reducing the need for complex joint optimization and enabling more efficient use of teacher models and RL compute.
Future work may explore automated switching criteria based on OPD validation scores and further investigate why OPD initialization outperforms SFT for RLVR. The two-stage approach could become a standard baseline for combining distillation and RL in reasoning model training.