MEDEC: Benchmark for Medical Error Detection and Correction in Clinical Notes, LLMs Still Lag Behind Doctors
In December 2024, Microsoft Research and others released the MEDEC benchmark, comprising 488 clinical notes (3,848 sentences) from three US hospitals, covering five error types: diagnosis, management, treatment, medication, and causative microorganism. Evaluated models include o1-preview, GPT-4, Claude 3.5 Sonnet, and Gemini 2.0 Flash. Results show LLMs underperform human doctors in both error detection and correction.
MEDEC is the first public benchmark focused on medical error detection and correction in clinical notes, filling a gap in evaluating LLMs' medical text verification capabilities. Experiments show that even the most advanced LLMs (e.g., o1-preview) lag behind human doctors in identifying and correcting medical errors, highlighting challenges in safety and reliability for clinical AI applications. The benchmark provides a standardized evaluation tool for future research.
The MEDEC dataset was constructed by medical experts, collecting 488 clinical notes from three US hospital systems. Each note contains multiple sentences, some of which are injected with errors (or kept correct). The five error types cover diagnosis (e.g., misdiagnosis), management (e.g., missed tests), treatment (e.g., wrong plan), medication (e.g., dosage error), and causative microorganism (e.g., wrong pathogen). Evaluation tasks include error detection (binary classification) and error correction (generating correct text). Metrics include exact match and ROUGE-L. Results show human doctors achieve an F1 of 0.85 on detection, while the best model o1-preview achieves 0.72; on correction, human doctors achieve 0.78 accuracy, o1-preview 0.65. Models perform worse on rare error types.
MEDEC provides important guidance for the medical AI industry: LLMs should be used cautiously in assisted diagnosis and clinical decision support, especially not directly for automatic correction of medical text. This benchmark can help hospitals and AI vendors evaluate models' actual capabilities in medical scenarios, driving the development of safer clinical AI systems.
It is recommended that healthcare IT companies and EHR vendors adopt MEDEC as a standard test for evaluating LLM clinical safety. Before deploying AI-assisted diagnostic systems, models should meet a threshold comparable to human doctors on error detection tasks. For institutions investing in medical AI, model performance on MEDEC should be considered a technology maturity indicator.
Future research may explore whether domain fine-tuning, retrieval augmentation, or human feedback can improve LLM performance on medical error detection. Expanding the MEDEC benchmark (e.g., adding more hospitals, languages, error types) will enhance its representativeness. Additionally, this work may lead to clinical AI verification tools for automatically detecting potential errors in electronic health records.