MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies
In April 2024, the paper released the MiniCPM series of small language models (1.2B and 2.4B non-embedding parameters), achieving SOTA at their respective scales, with performance comparable to 7B-13B models. It proposes the Warmup-Stable-Decay (WSD) learning rate scheduler, supporting continual training and domain adaptation. Through WSD, it discovers that the compute-optimal data-to-model ratio in the data-model scaling law is higher than the Chinchilla optimal. The series includes MiniCPM-DPO, MiniCPM-MoE, and MiniCPM-128K.
MiniCPM demonstrates that small language models (<3B parameters) can achieve performance comparable to large models (7B-13B) through carefully designed training strategies, challenging the 'bigger is better' consensus. The WSD learning rate scheduler enables efficient study of scaling laws and reveals that the compute-optimal data quantity is larger than suggested by Chinchilla. This has significant implications for model deployment costs, edge computing, and domain customization: small models can run on consumer-grade hardware with lower training costs. The open-sourcing of the MiniCPM series further promotes the SLM ecosystem.
MiniCPM adopts a Transformer architecture with 1.2B and 2.4B non-embedding parameters. The key innovation is the WSD learning rate scheduler, which consists of three phases: Warmup, Stable, and Decay. The stable phase maintains a high learning rate, while the decay phase rapidly reduces it, allowing the model to converge to a better local optimum. WSD supports stopping and evaluating at any point before the decay phase, enabling the study of scaling laws without retraining. Experiments show that under the same compute budget, MiniCPM-2.4B outperforms Mistral-7B, Llama-2-7B, etc., on multiple benchmarks. Limitations: Non-embedding parameter count excludes the word embedding layer, so total parameters are larger; performance evaluation is mainly based on English and Chinese benchmarks.
The MiniCPM series has a direct impact on AI deployment: 1) Reduces inference costs, enabling operation on phones and IoT devices; 2) Reduces dependence on high-end GPUs, accelerating AI democratization; 3) The MoE variant further compresses computation. For cloud service providers, small models can lower per-token costs, attracting price-sensitive customers. For the open-source community, MiniCPM provides a high-quality small model baseline, promoting fine-tuning in vertical domains.
It is recommended that edge computing and mobile teams evaluate MiniCPM-2.4B on target hardware for accuracy, time-to-first-token, memory, and energy consumption before deciding whether to replace larger models. Cloud service providers can verify hosted inference demand, while enterprises should first use domain regression sets to validate fine-tuning effectiveness.
Areas to watch: 1) MiniCPM's performance on multilingual and long-context tasks; 2) Generalizability of the WSD scheduler to models of other scales; 3) Ceiling of small models on complex reasoning tasks; 4) Combination with compression techniques like quantization and pruning. If small models continue to approach the performance of large models, it will change model selection strategies.