Don't Drop Dropout: Optimizing Layer Sparsity for Efficient LLM Training and Inference
A study shows layer dropout (stochastic depth) should be used in state-of-the-art LLM training. With optimal layer distribution, time schedule, and optimizer hyperparameters, layer dropout leads to lower loss at the same training FLOPs. For a given number of training steps, LLMs can achieve lower or similar validation loss while saving up to 25% of training FLOPs. Layer dropout enables post-training optimizations such as early exit, intermediate-layer skipping, and self-speculative decoding, yielding up to 1.5x inference speedup with negligible accuracy loss.
Layer dropout, also known as stochastic depth, has been shown to enable faster training, higher accuracy, and robustness to zero-shot layer pruning in both language and vision transformers. However, as models and datasets have scaled, dropout—particularly layer dropout—has largely disappeared from large language models (LLMs) pre-training recipes. While some prior work has reported that dropout can degrade accuracy, no comprehensive study has quantified, let alone mitigated, this effect. This study shows that layer dropout should be used in state-of-the-art LLM training, establishing best practices and scaling analysis for both training and post-training benefits. With optimal layer distribution, time schedule, and optimizer hyperparameters, layer dropout leads to lower loss at the same training FLOPs. For a given number of training steps, LLMs can achieve lower or similar validation loss while saving up to 25% of training FLOPs. Moreover, layer dropout enables significant post-training optimizations, such as early exit, intermediate-layer skipping, and self-speculative decoding, yielding up to 1.5x inference speedup with negligible accuracy loss.
The paper establishes that layer dropout, when properly configured with optimal layer distribution, time schedule, and optimizer hyperparameters, can reduce training FLOPs by up to 25% without sacrificing validation loss. It also enables post-training techniques like early exit, intermediate-layer skipping, and self-speculative decoding, achieving up to 1.5x inference speedup with negligible accuracy loss. This suggests a systematic approach to reintroducing stochastic depth in LLM pre-training, potentially altering standard training recipes.
If adopted, these techniques could significantly reduce the computational cost of training and serving large language models, making advanced AI more accessible and cost-effective. The 25% training FLOPs savings and 1.5x inference speedup could lower barriers for smaller players and accelerate iteration cycles. However, widespread adoption may require validation across diverse model architectures and datasets, and integration into existing training frameworks.
The potential to cut training compute by 25% and speed up inference by 1.5x translates directly into lower cloud costs and faster time-to-market for AI products. This could improve margins for AI service providers and make LLM-based applications more economically viable, especially for latency-sensitive or high-volume use cases.
Expect follow-up research to validate these findings on larger models and diverse datasets, and to explore the interaction of layer dropout with other efficiency techniques like mixture-of-experts and quantization. Tooling support in popular frameworks (e.g., PyTorch, JAX) may emerge. If results hold, layer dropout could become a standard component of LLM pre-training, leading to more efficient models and enabling new deployment scenarios.