Beyond Teacher Likelihood: Group-Calibrated On-Policy Distillation for Long-Context Reasoning
On-policy distillation (OPD) trains a student on its own responses using dense token-level guidance from a stronger teacher. In long-context tasks, token-level teacher support can favor locally plausible responses that omit evidence distributed across the input or violate global task constraints. Task-specific verifiers evaluate task completion at the response level and may return graded rewards that reflect partial success. Across longer input ranges, trajectory-level OPD scores become progressively less aligned with verifier rewards, indicating teacher-verifier disagreement. Group-Calibrated On-Policy Distillation (GC-OPD) separately normalizes verifier rewards and trajectory-level OPD scores within each rollout group and uses their difference as a signed teacher-verifier disagreement residual. Relative-advantage-based credit assignment (RACA) distributes this trajectory-level residual across tokens according to their relative OPD advantages while preserving the original OPD signal.
A research paper introduces Group-Calibrated On-Policy Distillation (GC-OPD) to address teacher-verifier disagreement in long-context reasoning tasks. The method normalizes verifier rewards and trajectory-level OPD scores within rollout groups, computes a signed disagreement residual, and uses relative-advantage-based credit assignment (RACA) to distribute the residual across tokens while preserving the original OPD signal.
GC-OPD introduces a group-level calibration mechanism that aligns token-level distillation signals with response-level verifier rewards. By computing a signed teacher-verifier disagreement residual and distributing it via relative OPD advantages, the method aims to correct locally plausible but globally incorrect token guidance in long-context tasks.
This work addresses a practical limitation in distilling large language models for long-context applications, where token-level teacher feedback may conflict with task-level success criteria. Improved alignment between distillation signals and verifier rewards could enhance the reliability of distilled models in evidence-aggregation and constraint-following tasks.
The method may reduce the need for expensive teacher supervision in long-context tasks by enabling more effective distillation with verifier feedback, potentially lowering training costs and improving model performance in enterprise applications requiring long-document understanding.
If validated across diverse long-context benchmarks, GC-OPD could become a standard component in distillation pipelines for reasoning-heavy models. Future work may explore its interaction with different verifier types and its scalability to very long contexts.