AxQM: A Textbook-Scale Benchmark for Formal Proof Synthesis in a Library of Finite-Dimensional Quantum Mechanics
AxQM is a benchmark of 1,019 kernel-checkable proof-synthesis tasks over 479 items drawn from the textbook Quantum Computation and Quantum Information by Nielsen and Chuang. The tasks are stated in a custom Lean library of finite-dimensional quantum mechanics. By task count, it is the largest proof-synthesis benchmark in physics by a factor of four. AxQM is derived from a near-complete formalization of the formal portions of the textbook, so every task is guaranteed a solution, which is kept private. Grading is done deterministically by the Lean kernel, which checks that the proof compiles and that no sorry appears in it or in any declaration it depends on.
Researchers released AxQM, a benchmark for evaluating autoformalization systems in physics. It contains 1,019 kernel-checkable proof-synthesis tasks over 479 items from Nielsen and Chuang's Quantum Computation and Quantum Information, stated in a custom Lean library of finite-dimensional quantum mechanics. It is the largest proof-synthesis benchmark in physics by task count, four times larger than previous benchmarks. Every task has a guaranteed solution, kept private, and grading is deterministic via the Lean kernel.
The benchmark leverages a custom Lean library for finite-dimensional quantum mechanics, enabling formalization of textbook-level physics arguments. The use of kernel-checkable tasks ensures rigorous verification, and the private solutions prevent leakage. The scale (1,019 tasks) suggests significant coverage of the formal portions of the textbook, providing a robust testbed for LLM-based autoformalization.
This benchmark addresses a gap in evaluating AI systems for formalizing physics, extending the rigor of proof assistants beyond mathematics. It may drive development of LLMs capable of handling domain-specific formalization, with potential applications in verification of physical theories and quantum computing.
The benchmark provides a standardized evaluation tool for companies developing AI for formal verification and scientific reasoning. It could lower barriers to adopting proof assistants in physics and quantum computing, potentially creating demand for specialized AI tools.
Expect follow-up benchmarks in other physics domains and increased use of Lean for physics formalization. The benchmark could become a standard for evaluating autoformalization in physics, and results may inform improvements in LLM reasoning for formal systems.