Soft Guidance Starts to Outperform CoT Prompting as LLMs Improve
A study evaluates modern mid-sized language models on math problem-solving (GSM8K) and finds that models specialized for reasoning achieve better performance in a zero-shot setting than with few-shot Chain-of-Thought (CoT) prompting. For example, Mathstral's accuracy improves from ~77% to ~84%. The authors attribute this to a 'guidance-distraction' tradeoff, where standard CoT prompting demands style adaptation and formatting compliance that can distract from core reasoning.
Chain-of-Thought (CoT) prompting has been a standard baseline for evaluating reasoning in large language models. However, as models improve, they often produce step-by-step reasoning natively. A new paper shows that for modern mid-sized reasoning-specialized models, zero-shot prompting outperforms few-shot CoT on math tasks, with Mathstral jumping from ~77% to ~84% on GSM8K. Even a general-purpose model benefits from zero-shot CoT over few-shot. The researchers suggest that CoT's extra demands for style and format can distract from reasoning, indicating that simpler 'soft guidance' may be more effective as LLMs advance.
The paper introduces a 'guidance-distraction' tradeoff: few-shot CoT examples impose formatting and stylistic constraints that can interfere with a model's native reasoning process. As models become more capable, they require less explicit step-by-step instruction, and zero-shot prompts allow them to leverage their internal reasoning abilities without the overhead of adapting to provided examples.
This finding could simplify prompt engineering for reasoning tasks, reducing the need for carefully crafted few-shot examples. It suggests that as models improve, the cost and complexity of prompting may decrease, making advanced reasoning more accessible and efficient for developers and enterprises.
Reduced reliance on few-shot prompting can lower inference costs and latency, as prompts become shorter. This makes reasoning tasks more scalable and cost-effective for commercial applications, potentially accelerating adoption in areas like automated math tutoring, code generation, and data analysis.
Future research may explore the boundaries of this effect across different model sizes, architectures, and reasoning domains. If the trend holds, we may see a shift toward minimal prompting strategies, with models relying more on pre-training and fine-tuning for reasoning capabilities. Watch for updates to benchmarking practices and prompt engineering best practices.