PAIChecker: Uncovering and Checking PR-Issue Misalignment in SWE-Bench-Like Benchmarks
A study of SWE-bench Verified instances found that 13.6% exhibit PR-issue misalignment across five patterns in eleven fine-grained scenarios. The authors propose PAIChecker, a multi-agent system for checking such misalignment, which achieves the best performance on SWE-Gym and SWE-bench Multilingual.
SWE-bench-like benchmarks pair pull requests with linked issues to evaluate LLM issue resolution. However, misalignment between PRs and issues is common. A systematic study of SWE-bench Verified reveals 13.6% misalignment across five patterns. To address this, PAIChecker, a multi-agent system with a three-phase design (pattern identification, cross-agent label synthesis, code-level validation), is introduced. It achieves top performance on SWE-Gym and SWE-bench Multilingual, enabling more reliable benchmark construction.
PAIChecker's three-phase design—specific pattern identification, cross-agent label synthesis, and code-level validation—enables accurate and generalizable detection of PR-issue misalignment, improving benchmark reliability.
Misalignment in widely used benchmarks like SWE-bench can lead to unreliable LLM evaluations. Tools like PAIChecker can improve benchmark quality, potentially influencing how organizations assess and select AI coding assistants.
Improved benchmark reliability can lead to better-informed decisions in AI tool procurement and development, reducing risks associated with flawed evaluations.
Adoption of PAIChecker or similar validation tools may become standard in benchmark creation, leading to more trustworthy evaluations. Future work could extend the approach to other benchmark types or integrate it into continuous integration pipelines.