Grouping the Stochastic Machine: Precision, Not Capability, as the Frontier Metric for AI Systems
An arXiv paper argues that frontier language models have saturated accuracy and that precision—the consistency of outputs across repeated identical requests—is the key differentiator. It proposes measuring precision by running a fixed suite of deterministically scored tasks multiple times at fixed temperature and computing per-task consistency, without using a model-in-the-loop grader.
The paper claims that benchmark culture focuses on central tendency (capability) rather than spread (precision), and that precision is measurable cheaply and without circularity. It further argues that precision measurement separates consistent failures (tight group off-centre, correctable by a sight adjustment) from scattered failures, guiding operational decisions.
The proposed method uses deterministic scoring on repeated runs at fixed temperature to compute per-task consistency, avoiding model-based evaluation. This provides a direct measure of output variance, which is currently absent from standard benchmarks.
If adopted, precision metrics could shift model selection and marketing from best-case capability to reliability, affecting how enterprises evaluate and deploy AI systems.
Precision measurement could help businesses identify models with consistent performance, reducing risk in production deployments and enabling targeted fixes for systematic errors.
Watch for follow-up work implementing the proposed precision benchmark suite and for benchmark providers adding variance metrics to leaderboards.