MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark
In June 2024, the MMLU-Pro team released an enhanced version of the multi-task language understanding benchmark, increasing options from 4 to 10 and removing trivial and noisy questions. Experiments show that model accuracy on MMLU-Pro drops by 16%-33% compared to MMLU, and sensitivity to 24 prompt styles decreases from 4-5% to 2%. Chain-of-thought reasoning outperforms direct answering on MMLU-Pro, while the opposite holds on MMLU.
MMLU-Pro addresses the performance saturation of the original MMLU by increasing reasoning difficulty and the number of options, becoming a more robust model evaluation tool. It reveals true gaps in complex reasoning and reduces the impact of prompt engineering, making it crucial for LLM capability tracking and competitive landscape analysis.
MMLU-Pro selects questions from MMLU that require more reasoning and expands options to 10, reducing guessing probability. Evaluation across 24 prompt styles shows that the standard deviation of model scores drops from 4-5% on MMLU to 2%, indicating improved robustness to prompts. Chain-of-thought reasoning improves by an average of 3-5% on MMLU-Pro, while it decreases on MMLU, proving that the new benchmark includes more reasoning steps. Limitations: still primarily English, and knowledge coverage may be biased toward the West.
MMLU-Pro is set to become a new standard for LLM evaluation, affecting performance claims in model releases. For AI companies, re-ranking on the new benchmark may change market perception. For downstream applications, more accurate evaluation helps select models suitable for complex reasoning tasks.
Recommend AI companies to immediately test their models on MMLU-Pro and publish results to build trust. Investment institutions can use MMLU-Pro as a core metric for model selection, prioritizing teams that perform well on this benchmark.
Key points to watch: 1) updates on mainstream model performance on MMLU-Pro; 2) whether it is adopted into official evaluation systems (e.g., OpenAI, Google); 3) expansion to multilingual versions; 4) feedback on model training strategies (e.g., reasoning enhancement).