arXiv · Jul 17, 2026
Understanding Reasoning from Pretraining to Post-Training
A study uses chess as a controlled testbed to examine how pretraining choices affect reinforcement learning (RL) post-training for reasoning. Language models from 5M to 1B parameters are pretrained on human chess games, fine-tuned on synthetic reasoning traces, and then RL-trained on chess puzzles with verifiable rewards. The post-RL performance at a given RL compute level is well-predicted by pretraining loss, and the slope of RL reward curves improves with model scale.
What happened
Researchers investigate the relationship between pretraining and RL post-training for reasoning by using chess as a controlled environment. They pretrain language models of varying sizes on human chess games, apply supervised fine-tuning on synthetic reasoning traces, and then perform RL on chess puzzles. Key findings include that pretraining loss strongly predicts post-RL performance, and larger models exhibit steeper improvements in RL reward curves.
Technical significance
The study demonstrates that pretraining loss serves as a reliable predictor of downstream RL performance, suggesting that pretraining quality directly influences the effectiveness of RL fine-tuning. The observed improvement in RL reward curve slopes with larger models indicates that scale enhances the model's ability to benefit from RL, potentially due to better learned representations during pretraining.
Industry impact
This research implies that investments in pretraining data quality and model scale can yield compounding returns during RL post-training, which is critical for developing advanced reasoning capabilities in AI systems. Companies may need to balance compute allocation between pretraining and RL to optimize overall performance.
What to watch
Future work may extend this framework to other reasoning domains beyond chess, such as mathematics or code generation, to validate the generalizability of these findings. Additionally, exploring the interaction between pretraining data diversity and RL efficiency could inform more cost-effective training pipelines.
Decision value
For AI developers, understanding the pretraining-RL relationship can guide resource allocation, potentially reducing the need for expensive RL compute by improving pretraining. This could lower costs and accelerate the deployment of reasoning-capable models in applications like automated problem-solving and strategic planning.