Small Batch Size Training for Language Models: Small-Batch Training Overturns Common Sense in Large Model Training, Gradient Accumulation Proven Wasteful
In July 2025, this paper proposes Adam hyperparameter scaling rules for small batch sizes (as low as batch size=1): keep the second moment half-life fixed in token dimension rather than step dimension. Experiments show small-batch training is stable, hyperparameter-robust, and achieves per-FLOP performance no worse than large-batch training, and supports stable training with momentum-free SGD. The authors recommend not using gradient accumulation unless multi-device multi-replica.
This research challenges the industry consensus that 'small-batch training is unstable' by rescaling Adam's β2 parameter (based on token half-life rather than steps), making batch size=1 training stable and efficient for language model pretraining and fine-tuning. Small batches not only save memory (no gradient accumulation) but also enable stable training with SGD (no momentum), significantly reducing optimizer state memory. This directly challenges the necessity of parameter-efficient fine-tuning methods like LoRA, as small batches with small-state optimizers can achieve full fine-tuning performance under similar memory.
Key finding: Adam's second moment decay rate β2 should be adjusted based on token half-life, i.e., β2 = exp(-ln(2) / (half_life_in_tokens / batch_size)). When batch size decreases, β2 should increase to maintain the same token half-life. Experiments pretrain 125M-1.5B parameter models on C4 dataset, validating convergence curves for batch sizes 1, 2, 4, 8, 16, 32, 64, 128. Small batches (≤8) are not inferior to large batches (≥64) in validation loss and are more robust to hyperparameter changes like learning rate and β1. With SGD (no momentum), batch size=1 still trains stably, while large-batch SGD diverges. Gradient accumulation is proven ineffective: under the same total batch size, increasing accumulation steps reduces optimization steps, degrading performance.
This result has profound implications for large model training infrastructure. Small-batch training reduces GPU memory requirements and distributed communication overhead (no gradient synchronization), enabling single-GPU training of larger models. Cloud service providers can offer more flexible training instance configurations. The market value of fine-tuning methods like LoRA may be weakened, as full fine-tuning can be achieved under similar memory. Training frameworks (e.g., PyTorch FSDP, DeepSpeed) need to optimize computational efficiency for small-batch scenarios.
It is recommended that AI training platforms (e.g., Hugging Face, Weights & Biases) integrate small-batch training best practices into default configurations. GPU cloud service providers can launch small-batch optimized instances to reduce user costs. Model fine-tuning service providers should reassess the cost-effectiveness of LoRA vs full fine-tuning. Internal enterprise training teams can immediately adopt small batches with small-state optimizers (e.g., Adafactor) to reduce memory footprint.
Focus on validation of small-batch training on larger models (>10B) and stability in long-sequence training. Need to test applicability under different architectures (e.g., MoE, Mamba). In terms of safety, small batches may introduce higher gradient noise, requiring study of impact on model robustness. Cost-wise, small batches can lower hardware barriers but need to trade off computational efficiency (small-batch GPU utilization may decrease). If adopted by mainstream frameworks, it will change training practices.