From Static to Dynamic: Benchmarking Real-World Code Review with MCR-Bench
MCR-Bench is introduced as the first defect state-aware benchmark for realistic multi-round code review. It covers five programming languages and consists of 2,269 real-world multi-round code review tasks, each annotated with fine-grained defect information and cross-round state labels. Extensive experiments were conducted on MCR-Bench with mainstream LLMs.
A new benchmark, MCR-Bench, addresses the gap between simplified single-round code review tasks and real-world iterative code review. It includes 2,269 tasks across five programming languages, with fine-grained defect metadata and dynamic state annotations. Experiments with mainstream LLMs reveal limited overall performance, highlighting challenges in multi-round code review.
The benchmark's defect state-aware design enables evaluation of LLMs' ability to track defect evolution across rounds. Initial findings suggest current LLMs struggle with the dynamic, multi-round nature of code review, indicating a need for models that can maintain context and adapt to changing defect states.
Automated code review tools may need to shift from single-pass analysis to interactive, multi-round workflows to better support real development processes. This benchmark could drive development of more sophisticated LLM-based review assistants.
For companies building AI code review tools, MCR-Bench provides a more realistic evaluation standard, helping differentiate products that handle iterative review. It may also reduce development costs by identifying model weaknesses early.
Expect follow-up research using MCR-Bench to improve LLM performance on multi-round code review, potentially leading to new training paradigms or architectures that handle iterative feedback. Adoption of such benchmarks may become standard for evaluating code review AI.