LLM Judges Verify Presence, Not Absence: Omission Blindness in AI Clinical Notes and What Recovers It
A benchmark of 500 single-error note pairs (298 with a named fact certainly absent, 202 added-or-altered controls) found that across eight LLM judge designs, paired discrimination on omissions was 0.50-0.63 (near coin flip), while on added or altered content it was 0.79-0.94. On single notes, no design flagged omissions reliably more often than perfect notes. Restructuring the task to list transcript facts then check the note recovered detection via a per-fact pipeline and a GEPA-evolved prompt.
Ambient AI scribes draft clinical notes, and published audits find their dominant error is omission. The standard check is an LLM judge: a second model reads the note against the transcript and flags problems. The study asks whether judges detect omissions. Public corpora cannot supply the answer key because clinician reference notes and transcripts are materially discrepant. The benchmark has 500 single-error note pairs from audited fact sheets, 298 with a named fact certainly absent and 202 added-or-altered controls. Across eight judge designs, paired discrimination (flawed note below its clean twin, 0.5 a coin flip) reads 0.79-0.94 on added or altered content and 0.50-0.63 on omissions. On single notes, no design flags omissions reliably more often than perfect notes. Wording changes, voting and GEPA prompt optimisation move the operating point without creating usable detection. Restructuring the task recovers it: list the facts the transcript establishes, then check the note for each. Two methods reach it independently and trade off: a per-fact pipeline, and a GEPA-evolved prompt doing the same in one call.
LLM judges exhibit omission blindness: they verify presence of facts but fail to detect absence. Paired discrimination on omissions is near chance (0.50-0.63) versus high on added/altered content (0.79-0.94). Restructuring the evaluation from direct note-vs-transcript comparison to explicit fact enumeration followed by per-fact verification recovers detection, with a per-fact pipeline and a GEPA-evolved single-call prompt as two independent methods.
For AI scribe vendors and healthcare systems, relying on LLM judges to audit clinical notes for omissions is unsafe with current prompting. The finding implies that quality assurance workflows must be redesigned to explicitly enumerate expected facts before checking, or they will systematically miss the dominant error class. This has direct implications for regulatory compliance and patient safety in ambient documentation products.
The research identifies a critical failure mode in LLM-based quality assurance for AI clinical documentation. Companies that implement the recovered detection methods (per-fact pipeline or GEPA-evolved prompt) can differentiate by offering more reliable omission detection, reducing risk of missed clinical information and potential liability. This creates an opportunity for evaluation tooling vendors and a competitive threat to scribe products relying on naive LLM judges.
Next observable signals include adoption of fact-enumeration-based auditing in clinical AI scribe products, publication of follow-up benchmarks testing omission detection in other domains (legal, financial), and potential integration of GEPA-optimized prompts into commercial evaluation pipelines. If omission blindness generalizes, expect a shift in evaluation methodology from pairwise judge scores to structured fact-checking protocols.