AIDev-pop: Real Performance of Coding Agents Requires Simultaneous Assessment of Merge, Review, and Commit Quality
A study submitted on February 2, 2026, used AIDev-pop, which includes thousands of open-source PRs, to compare five types of coding agents. It found that Codex has a higher merge rate, Copilot triggers the most review discussions, and commit message quality is not synchronized with merge results.
A single SWE benchmark score cannot explain how agents enter real collaboration. This study compares task acceptance, review communication, and commit quality across the PR lifecycle, revealing independent advantages in integration and engineering expression among different products.
The evaluation extracts AI-generated PRs from public open-source repositories and compares acceptance rates, human and automated review discussion volume, and commit message quality by task type. Results suggest these dimensions are not collinear: high merge rates may accompany weaker commit messages, and more review discussions do not necessarily indicate low quality.
Team selection of code agents will shift from model rankings to alignment with existing review, CI, and commit conventions; platforms must retain granular data on task types and review burden.
Enterprises should track merge rate, review time, rework, and defects by task category during pilots, rather than only counting generated code volume or completed PRs.
Need to control for repository, task difficulty, and user differences, and continue to validate PR follow-up defects, rework, maintenance costs, and author selection bias.