RISE: Recursive Improvement via Self-Extrapolating Policy Distillation
RISE (Recursive Improvement via Self-Extrapolating Policy Distillation) is proposed as a method that constructs a synthetic teacher from a model's own RLVR training trajectory by extrapolating the displacement between the current checkpoint and a trailing anchor in parameter space or output logit space. It converts a sparse outcome-induced parameter update into a dense token-level target without external models or privileged conditioning. RISE combines RLVR and on-policy distillation in a complementary loop, with the teacher refreshed every iteration. The paper reports experiments spanning mathematical reasoning tasks.
The paper introduces RISE, a self-improvement technique for language model post-training that uses the model's own RLVR trajectory to create a dense token-level distillation target. By extrapolating parameter or logit displacement between checkpoints, RISE avoids external teachers and privileged conditioning, enabling recursive improvement as the student model advances.
RISE leverages the difference between a current checkpoint and a trailing anchor from RLVR training to generate a synthetic teacher. This teacher provides dense per-token supervision, addressing distribution mismatch and in-context learning limits of prior on-policy distillation methods. The recursive refresh of the teacher each iteration turns distillation into a continuous improvement loop rather than a one-shot compression.
This approach could reduce reliance on large external teacher models for post-training, potentially lowering costs and simplifying pipelines for model developers. It may enable more efficient self-improvement cycles in production LLMs, particularly for reasoning-heavy tasks.
RISE offers a potential cost and complexity reduction in LLM post-training by eliminating external teacher models and privileged conditioning. It could improve model performance on reasoning tasks through recursive self-distillation, enhancing product capabilities for enterprises and developers.
Observable next signals include publication of detailed experimental results on mathematical reasoning benchmarks, open-source code or model checkpoints, and adoption or replication by major AI labs. Further research may explore scaling RISE to larger models and diverse domains.