Symbolic Discovery of Optimization Algorithms (Lion): Symbolic Search Discovers Efficient Optimizer Lion, Drastically Reducing Training Cost
In February 2023, Google proposed the Lion optimizer, automatically discovered through symbolic program search. Lion only tracks momentum and uses the sign function to update parameters, making it more memory-efficient than Adam. On ImageNet, ViT accuracy improved by 2%, and JFT pretraining computation was reduced by 5x; diffusion model training computation was reduced by 2.3x. It has been deployed in Google Search ad CTR models.
Lion was automatically discovered through program search rather than manual design, demonstrating the feasibility of automated algorithm discovery. Its simplicity (only requiring momentum) and efficiency (less memory, faster computation) make it a strong alternative to Adam. It outperforms Adam on multiple tasks, especially in large-batch training scenarios, and is expected to become the new default optimizer for deep learning training.
Lion's update rule is: θ_t = θ_{t-1} - lr * sign(β1 * m_{t-1} + (1-β1) * g_t), where m_t = β2 * m_{t-1} + (1-β2) * g_t. Compared to Adam, Lion removes second-moment estimation and bias correction, maintaining only first-order momentum, and the update magnitude is determined by the sign function (each parameter has the same update magnitude). Search method: uses evolutionary algorithms to search for update rules on proxy tasks (e.g., small CNN training) and then transfers to large-scale tasks. Evaluation covers image classification (ViT, ResNet), vision-language contrastive learning (CLIP), diffusion models, and language models (BERT, GPT). Key findings: Lion's performance gain increases with batch size; requires a smaller learning rate than Adam (about 1/10). Limitations: improvement is not significant on some small-batch tasks; sign updates may lead to overly aggressive parameter update directions.
Lion reduces the computational and memory costs of deep learning training, especially benefiting large-scale model training. For cloud service providers, this means lower training costs and higher resource utilization. For AI companies, adopting Lion can shorten model iteration cycles and reduce hardware investment. Its automatic discovery method also provides a paradigm for automatic design of other algorithms (e.g., learning rate schedules, data augmentation).
AI training platforms can integrate Lion as the default optimizer to reduce user training costs. Enterprises can evaluate replacing Adam with Lion in their own model training, expecting 20-50% training time savings. It is recommended to prioritize trials on vision models and diffusion models.
Focus on Lion's validation on larger-scale models (e.g., LLMs) and exploration of its variants (e.g., combining adaptive learning rates). Community adoption and native support in frameworks like TensorFlow/PyTorch need to be observed. Automatic algorithm discovery methods may extend to other domains.