Large Language Monkeys: Scaling Inference Compute with Repeated Sampling
In July 2024, research from Anthropic and other institutions found that repeated sampling (generating multiple candidate answers) can significantly improve LLM problem-solving coverage. On SWE-bench Lite, DeepSeek-Coder-V2-Instruct improved from 15.9% with single sampling to 56% with 250 samples, surpassing the single-sample SOTA of 43%. Coverage follows a log-linear relationship with sampling count, modeled by an exponential power law.
This work reveals the potential of scaling inference compute: through simple repeated sampling, LLM performance on automatically verifiable tasks (e.g., programming, math) can be greatly improved. It challenges the paradigm of focusing only on training-time scaling, showing that inference-time compute investment is equally important. For tasks without automatic verification, majority voting and reward models saturate after hundreds of samples, suggesting the need for better selection methods.
The study tested repeated sampling on multiple tasks and models, finding a log-linear relationship between coverage and sampling count, fitted by an exponential power law. On SWE-bench Lite, 250 samples achieved 56% resolution for DeepSeek-Coder-V2-Instruct, exceeding single-sample SOTA. For tasks without automatic verification, majority voting and reward models plateau after hundreds of samples.
These results have direct implications for products like AI coding assistants and math solvers: increasing inference compute (e.g., multiple generations with verification) can significantly boost performance. It also drives demand for inference infrastructure optimization.
Recommend that teams building AI coding tools and math solvers immediately deploy repeated sampling strategies and optimize verification pipelines. Consider dynamically adjusting sampling count based on task difficulty to balance cost and effectiveness.
Focus on more efficient candidate answer selection methods (e.g., process reward models) and the synergy between inference-time and training-time scaling. Expect hardware and scheduling systems optimized for inference.