Blog: Survey of Optimizers
A survey published on arXiv on 2026-08-28 organizes recent neural-network optimizers along four axes: temporal estimation, update geometry, horizon management, and representation and systems. It covers Muon, Shampoo, SOAP, adaptive and hybrid matrix methods, memory-efficient optimizers, schedule-free training, small-batch corrections, and quantized optimizer states. The survey concludes that matrix-aware methods are a genuine advance but there is no context-independent replacement for AdamW.
The survey argues that neural-network optimization in 2025-2026 is no longer a succession of new Adam variants. The design space has expanded from coordinates to matrices and layers, from fixed training horizons to policies over time, and from mathematical update rules to state representations that must survive sharding and low-precision computation. It connects spectral normalization of Muon, historical matrix statistics of Shampoo and SOAP, adaptive and hybrid matrix methods, memory-efficient optimizers, schedule-free training, small-batch corrections, and quantized optimizer states. The central empirical conclusion is that matrix-aware methods represent a genuine advance, but there is no context-independent replacement for AdamW. Rankings change with model scale, data-to-parameter ratio, batch size, schedule, parameter partition, tuning budget, and whether the target metric is tokens, FLOPs, wall-clock time, or memory.
The survey identifies four largely independent axes for optimizer design: temporal estimation, update geometry, horizon management, and representation and systems. It highlights that optimizer state representations must survive sharding and low-precision computation, indicating a shift toward systems-aware optimizer design. Matrix-aware methods such as Muon, Shampoo, and SOAP are presented as a genuine advance over coordinate-wise methods like AdamW, but their benefits are context-dependent.
The practical consequence is a compositional approach to optimizer selection: practitioners should choose components along each axis based on their specific training constraints rather than expecting a single drop-in replacement for AdamW. This suggests that optimizer benchmarking and selection will become more nuanced, with no universal best choice across model scales, data regimes, and hardware configurations.
For AI training teams, the survey provides a framework to reduce optimizer selection risk by decomposing the choice into independent axes. This can lead to more efficient use of compute and memory, potentially lowering training costs and time-to-model for large-scale models. However, the lack of a universal replacement for AdamW means teams must invest in tuning and evaluation for their specific workloads.
Observable next signals include: increased adoption of matrix-aware optimizers in large-scale training runs where memory and wall-clock efficiency are critical; development of hybrid optimizers that combine matrix statistics with schedule-free training; and more systematic benchmarking that reports optimizer performance across multiple axes (tokens, FLOPs, wall-clock, memory) rather than a single metric.