Adversarial Fast-Moving Real-World Domains as Test Beds for Benchmarking AI Scientist Capabilities
A paper proposes using adversarial, fast-moving real-world domains (Formula 1 and Magic: The Gathering) to benchmark AI scientist capabilities. In F1, GPT-5.2 matched 10 of 40 real innovations across 166 proposed ideas. In MTG, models proposed decks from a recently updated card pool and were evaluated against 19 Pro Tour decklists; the best deck performance is not fully specified in the evidence.
Benchmarking AI scientists' ability to generate novel ideas is challenging due to reliance on synthetic or retrospective tasks. This paper hypothesizes that complex, adversarial, fast-moving real-world domains can provide a practical solution. It instantiates this framework in Formula 1 (F1) and Magic: The Gathering (MTG). In F1, models ideated car design concepts for the 2026 season, with GPT-5.2 matching 10 of 40 real innovations across 166 proposed ideas. In MTG, models proposed decks from a recently updated card pool, evaluated against 19 Pro Tour decklists. The evidence indicates models produce plausible outputs but few align with real-world expert solutions.
The framework leverages domains with observable expert outputs and rapid evolution to reduce confounding from prior exposure. The use of adversarial, fast-moving environments tests reasoning, novelty, and hypothesis formulation under conditions closer to real scientific discovery. The limited alignment between model outputs and expert solutions suggests current models struggle with generating truly novel, practical ideas in complex domains.
This approach could influence how AI research capabilities are evaluated, shifting from static benchmarks to dynamic, real-world test beds. It highlights a gap between plausible generation and expert-level innovation, which may impact investment in AI for scientific discovery. The involvement of GPT-5.2 indicates leading models are being tested, but results show room for improvement.
For organizations investing in AI-driven R&D, this benchmark provides a more realistic measure of AI's ability to innovate. It could inform decisions on deploying AI in competitive, rapidly changing industries. The results indicate current AI may assist but not replace human experts in generating breakthrough ideas, affecting ROI calculations for AI scientist tools.
Next signals include whether other research groups adopt similar real-world adversarial benchmarks, and if model developers use such frameworks to guide improvements in scientific reasoning. Future work may expand to other fast-moving domains like cybersecurity or financial markets. The performance gap suggests a need for architectures that better handle novelty and adversarial conditions.