SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model
Submitted in February 2025. SmolLM2 is a 1.7B parameter small language model trained on approximately 11 trillion tokens through multi-stage training, mixing web text, math, code, and instruction data. Three new datasets are introduced: FineMath (math), Stack-Edu (code education), and SmolTalk (instructions). Through small-scale ablation and manual tuning of data mixture ratios at each stage, it surpasses Qwen2.5-1.5B and Llama3.2-1B on multiple benchmarks. The model and datasets are fully open-sourced.
SmolLM2 demonstrates the importance of data quality and training strategies for small models: through carefully designed multi-stage training and specialized datasets, a 1.7B parameter model can outperform larger or similarly sized competitors. This provides high-performance small model options for resource-constrained scenarios (e.g., mobile, edge devices), while open-sourced datasets can foster community research. Its data mixing approach (general first, then specialized) may become a paradigm for small model training.
SmolLM2 adopts three-stage training: Stage 1 pre-trains on approximately 9 trillion general web text tokens; Stage 2 continues training on 1 trillion math and code tokens (using FineMath and Stack-Edu); Stage 3 fine-tunes on 1 trillion instruction tokens (using SmolTalk). FineMath is built by filtering high-quality math web pages, Stack-Edu extracts educational content from Stack Exchange, and SmolTalk contains diverse instructions. Ablation experiments show that adding math and code data in Stage 2 improves reasoning ability by 15%, and instruction fine-tuning in Stage 3 improves dialogue ability by 20%. On benchmarks such as MMLU, GSM8K, and HumanEval, SmolLM2 achieves 45.2%, 52.3%, and 38.4% respectively, all higher than Qwen2.5-1.5B and Llama3.2-1B.
SmolLM2 will drive the deployment of small models on phones, IoT devices, and offline scenarios. Its open-source nature can reduce customization costs for enterprises. Specialized datasets (e.g., FineMath) can be reused by other models, accelerating small model R&D in math and code domains. It may influence the software stack design of edge AI chips.
Recommend mobile AI teams evaluate the possibility of replacing existing small models (e.g., Phi-2, TinyLlama) with SmolLM2, prioritizing testing in offline translation and intelligent customer service scenarios. Consider developing on-device AI applications based on SmolLM2, or using its open-source datasets to train proprietary models.
Focus on SmolLM2's inference speed and power consumption on mobile devices (e.g., Qualcomm, MediaTek chips). Whether its multi-stage training method will be adopted by other small models (e.g., TinyLlama, Phi series). Observe the quality of instruction datasets built by the community based on SmolTalk, and whether domain-specific fine-tuned versions based on SmolLM2 emerge.