Efficient Knowledge Distillation for LLMs: Offline Top-K Logits and a Fused Chunked KL Loss
A study on making knowledge distillation (KD) for large language models more efficient presents two systems contributions: offline KD with cached teacher top-K logits matches online distillation in training loss while removing the teacher from memory, running about 29% faster per iteration and achieving up to 41% higher throughput on a single H200 GPU; and a fused, chunked KL loss that avoids materializing the full vocabulary-sized logit tensor, making peak memory linear in sequence length and enabling training at 32,768 tokens context on a single GPU.
Researchers propose two methods to improve the efficiency of knowledge distillation for compressing large language models into smaller ones. Offline KD, which caches the teacher's top-K logits once and trains the student against this cache, matches online distillation performance while eliminating the need to keep the teacher model in memory, resulting in faster iterations and higher throughput. Additionally, a fused, chunked KL loss function reduces peak memory usage by avoiding the creation of the full vocabulary-sized logit tensor, allowing training with much longer context lengths on a single GPU.
The offline KD approach decouples teacher inference from student training, enabling significant speedups and memory savings without loss in distillation quality. The fused chunked KL loss addresses the memory bottleneck of the standard KL divergence computation by processing logits in chunks, which scales memory linearly with sequence length rather than vocabulary size, thus enabling longer context training on limited hardware.
These techniques lower the computational barrier for deploying compressed language models, making it more feasible for organizations with limited GPU resources to perform high-quality distillation. The ability to train with longer contexts on a single GPU could accelerate the adoption of small, efficient models in latency-sensitive and on-premises applications.
Reducing the cost and hardware requirements for knowledge distillation enables more companies to create custom small language models tailored to specific domains or deployment constraints, potentially expanding the market for on-device and private AI solutions.
If these methods are validated across diverse model architectures and tasks, they could become standard practices in model compression pipelines. Further work may explore combining offline KD with other compression techniques like quantization or pruning, and extending the chunked loss approach to other large-output-space problems.