One note in three: a verified census of three deployed AI scribes, and the instrument that counted it
An audit of three commercial ambient AI scribes on 142 consultations (565 notes) found 31.3% of notes contained verified failures, concentrated in allergy and medication information, invented patient identity, and history written as examination on telephone consultations. Excluding errors a patient record would have prefilled, the rate was 24.8%. Two clinicians adjudicated blind samples, upholding 20 of 21 and 12 of 12 findings.
Researchers audited three commercial AI scribes on the same 142 consultations, generating 565 notes from recorded UK primary-care and US ambulatory encounters plus authored scenarios. Twelve discovery passes proposed 13,678 candidate errors; 5,898 passed an importance filter and went to an adversarial panel of two models from different families, each told to refute what it could, leaving 618 verified failures. One note in three (31.3% [27.0, 35.6]) carried a verified failure, concentrated in allergy and medication information, invented patient identity, and history written as examination on telephone consultations that can contain none. No product was given a patient record; excluding invented identity and dates, the rate was 24.8% [20.8, 29.0]. A treatment the clinician retracts, recorded as delivered care, was a failure mode not in published taxonomies. Two clinicians adjudicated blind, disjoint samples: a physician author upheld 20 of 21 findings (95.2% [77.3, 99.2]) and an independent clinician, not an author, upheld 12 of 12.
The audit used a multi-stage pipeline: 12 discovery passes generated 13,678 candidate errors, an importance filter reduced them to 5,898, and an adversarial panel of two models from different families, each instructed to refute findings, left 618 verified failures. This adversarial verification approach aims to reduce false positives in error detection. The concentration of errors in allergy and medication information suggests current scribe models struggle with structured clinical data extraction and contextual reasoning, especially in telephone consultations where physical examination is impossible.
The finding that one in three AI-generated clinical notes contains a verified error, even without patient records, signals significant reliability gaps in ambient AI scribes. This may slow adoption in high-stakes clinical settings and increase demand for human-in-the-loop review, error detection tools, and integration with electronic health records to prefill known patient data. The study's methodology could become a benchmark for evaluating scribe accuracy.
The audit provides a quantitative benchmark for AI scribe reliability, useful for procurement decisions, risk assessment, and vendor differentiation. Companies that can demonstrate lower verified error rates, especially in allergy and medication fields, may gain competitive advantage. The methodology offers a reusable framework for quality assurance and could be licensed or adopted by healthcare systems.
Expect increased scrutiny from regulators and healthcare providers on AI scribe accuracy, potentially leading to mandatory error-rate disclosures and certification standards. Vendors may invest in domain-specific fine-tuning, retrieval-augmented generation from patient records, and real-time error detection. Future audits may expand to more products, languages, and specialties, and track error rates over time as models improve.