Rethinking On-Policy Distillation of Large Language Models II: One Training Example
A study examines on-policy distillation (OPD) at the data-minimal limit by training on a single query. One-shot OPD keeps improving for hundreds of steps and recovers most of full-data OPD's gain across task domains and model families. A single query reaches 71.5% state coverage, most within the first 100 steps. Adding semantically distinct queries raises coverage and validation accuracy together, until 16 queries reach 98.9% and match full-data training. Alignment slows at a similar pace whether OPD trains on one query or the whole dataset, and even a fixed set of states takes hundreds of steps to absorb. The paper concludes OPD is data-overfed but algorithm-starved.
On-policy distillation (OPD) combines student-generated rollouts with dense token-level supervision from a teacher. Existing work has mainly studied its algorithmic behavior, leaving the role of training data unclear. This paper examines that role at the data-minimal limit by training on a single query. One-shot OPD keeps improving for hundreds of steps and recovers most of full-data OPD's gain across task domains and model families. The authors explain this result through the states visited during training and the rate at which the student aligns with the teacher. They measure state coverage, the fraction of the states full-data OPD visits that a query set's rollouts reach. A single query already reaches 71.5%, most of it within the first 100 steps. Adding semantically distinct queries raises coverage and validation accuracy together, until 16 queries reach 98.9% and match full-data training. Yet alignment slows at a similar pace whether OPD trains on one query or the whole dataset, and even a fixed set of states takes hundreds of steps to absorb. OPD is therefore data-overfed but algorithm-starved. Its rollouts quickly expose broad supervision, while the student absorbs that supervision slowly.
The paper introduces a state coverage metric to quantify how much of the state space visited during full-data OPD is reached by rollouts from a limited query set. It finds that a single query already covers 71.5% of states, and 16 queries cover 98.9%, indicating that the diversity of states is quickly exposed. However, the student's alignment with the teacher improves at a similar slow rate regardless of data volume, suggesting that the bottleneck is the optimization algorithm rather than data availability. This implies that improving the learning algorithm could yield larger gains than collecting more data.
The finding that OPD is data-overfed but algorithm-starved suggests that practitioners may be over-investing in data collection for distillation. Instead, resources could be redirected toward developing better optimization techniques or more efficient training procedures. This could reduce the cost and time required to distill large language models, making the technology more accessible. The result also implies that small, carefully chosen query sets can be nearly as effective as full datasets, which is valuable for scenarios with limited data or privacy constraints.
This research indicates that companies can achieve most of the benefits of on-policy distillation with as few as 16 carefully selected queries, reducing data collection and annotation costs. It also highlights that current distillation processes may be inefficient, and investing in algorithmic improvements could yield better returns than scaling data. For AI service providers, this could lower the barrier to deploying distilled models, enabling more cost-effective offerings. The insights may also inform the design of more efficient training pipelines, potentially reducing compute and time-to-market.
Future research may focus on improving the optimization algorithms used in on-policy distillation to accelerate student alignment. The state coverage metric could become a standard tool for evaluating the sufficiency of training data in distillation and other imitation learning settings. If algorithmic improvements are achieved, the data requirements for distillation could be drastically reduced, leading to faster and cheaper model compression. Additionally, the observation that a fixed set of states takes hundreds of steps to absorb suggests potential for curriculum learning or replay strategies to enhance efficiency.