Clean Engineering, Unstable Measurement: A Preregistered Reliability Failure of Black-Box LLM Observers on Shared Endpoints
A preregistered audit of black-box LLM observers found that same-window repeat rankings agreed at Spearman 0.400 against a required 0.90, and byte-identical next-day replays agreed at 0.78 against a required 0.99, across 52,988 audited request attempts. Three mechanisms were identified: label-to-meaning mapping bias, candidate gaps seven orders of magnitude below the instrument's noise floor, and byte-identical inputs returning different rankings.
Language-model judges are used to gate training data, score generations, and drive leaderboards, but their reliability as measurement instruments is rarely verified. This study audited the assumption that the same request to the same model name yields consistent results. Two preregistered campaigns with fixed thresholds failed instrument validation. Same-window repeat rankings had Spearman correlation 0.400 (required 0.90), and byte-identical next-day replays had 0.78 (required 0.99), despite flawless execution. The paper identifies three mechanisms: biased label-to-meaning mapping, candidate gaps far below the noise floor, and non-deterministic outputs for identical inputs. Metric substitution and sampling did not fix the issue. Follow-ups showed waiting did not help (0.805 vs 0.800 over five days) and switching providers did not help (four providers share the issue).
The study reveals that black-box LLM observers exhibit significant non-determinism and bias, undermining their use as reliable measurement instruments. The Spearman correlations of 0.400 and 0.78 indicate poor ranking stability. The identified mechanisms suggest that the model's internal label-to-meaning mapping introduces systematic bias, and the noise floor is much larger than the candidate differences being measured. This implies that current LLM-based evaluation pipelines may produce unreliable rankings, especially when comparing models with similar performance.
The findings challenge the widespread industry practice of using LLM judges for model evaluation, data filtering, and leaderboard rankings. Companies relying on such judges for model selection or data curation may be making decisions based on unstable measurements. The failure to achieve reliability even with metric substitution or provider switching suggests a systemic issue across major LLM providers. This could lead to increased scrutiny of evaluation methodologies and a push for more deterministic or transparent judging approaches.
For businesses using LLM judges in production, this research highlights the risk of unreliable evaluations leading to poor model selection, biased data filtering, and misleading leaderboard positions. Investing in more robust evaluation infrastructure or adopting alternative measurement methods could prevent costly mistakes. The study also suggests that current LLM-as-a-judge services may not meet enterprise reliability requirements, creating an opportunity for vendors offering validated, deterministic judging solutions.
Observable next signals include: (1) increased adoption of deterministic or self-consistency techniques in LLM judging, (2) development of new evaluation benchmarks that account for judge variability, (3) potential industry standards for reporting judge reliability metrics, and (4) research into reducing label-to-meaning bias through prompt engineering or fine-tuning. The preregistered follow-ups indicate that simple fixes like waiting or switching providers are insufficient, so more fundamental changes are likely.