Who Thinks Best Depends on How Long You Let Them: Budget-Dependent Rankings in LLM Evaluation
A study evaluated four large language models on three reasoning benchmarks across seven token generation budgets (64 to 4,096 tokens), totaling 56,476 inferences. It found that 3–19% of items exhibit non-monotone accuracy with more budget, model rankings reverse across budgets on all benchmarks (p < 0.01, McNemar), oracle analysis shows model complementarity up to +27.8 percentage points, and a budget-aware router captures 14.1% of the oracle gap cross-domain.
Standard LLM evaluation assumes stable model rankings across inference conditions. This research challenges that assumption by varying token generation budgets across seven levels (64–4,096) for four models on three reasoning benchmarks (56,476 inferences). Key findings: 3–19% of items show non-monotone behavior (accuracy decreasing with more budget), model rankings reverse across budgets on all benchmarks (p < 0.01), oracle analysis reveals model complementarity up to +27.8pp, and a budget-aware router captures 14.1% of the oracle gap cross-domain. The authors argue for budget-conditioned evaluation protocols.
The study demonstrates that token generation budget is a critical variable in LLM evaluation, causing ranking reversals and non-monotone accuracy. Budget-aware routing can recover part of the oracle complementarity, but budget features are domain-specific and may hurt cross-domain transfer.
Evaluation practices that assume stable model rankings may mislead model selection and deployment decisions. Budget-conditioned evaluation protocols could become necessary for fair benchmarking and cost-effective model routing in production systems.
For AI service providers and enterprises, budget-aware model selection and routing can improve accuracy per token cost, potentially reducing inference expenses while maintaining or improving performance. This research highlights a new dimension for optimizing LLM deployments.
Expect increased adoption of budget-conditioned evaluation in benchmarks and model cards. Further research may focus on developing robust budget-aware routing methods that generalize across domains and on understanding the causes of non-monotone behavior.