SWE-Gate: Passing Functional Tests Is Not Enough for Software Engineering Agents
SWE-Gate is a repository-level benchmark for software engineering agents that evaluates review constraint compliance alongside functional correctness. It derives review constraints from real pull request review comments and synthesizes repository-level repair instances around these constraints. Each instance provides separate functional and constraint tests, together with non-compliant and gold patches. SWE-Gate contains 303 repository-level repair instances spanning 75 open-source Python repositories across diverse software domains. Experiments with four LLM backends spanning different capability levels under a common coding-agent scaffold reveal a substantial gap between functional success and review constraint compliance.
SWE-Gate is a new benchmark for software engineering agents that goes beyond functional test passing to include review-derived acceptance constraints. It contains 303 repair instances across 75 Python repositories, each with separate functional and constraint tests. Experiments with four LLM backends show a substantial gap between functional success and review constraint compliance, indicating that current agents may not meet real-world code review standards.
The benchmark separates functional correctness from review constraint compliance by providing distinct tests for each. This design enables precise measurement of an agent's ability to satisfy human-like code review feedback, not just pass unit tests. The observed gap suggests that LLM-based coding agents may require additional mechanisms, such as explicit constraint modeling or retrieval of review comments, to improve acceptance rates.
For organizations adopting AI coding agents, SWE-Gate highlights a critical limitation: agents that pass tests may still produce patches rejected by human reviewers. This could slow enterprise adoption unless tools incorporate review-aware training or post-generation filtering. The benchmark may become a standard for evaluating agent quality in software engineering workflows.
SWE-Gate provides a more realistic evaluation metric for coding agents, helping enterprises select tools that produce review-ready code. Vendors that improve compliance could differentiate in a crowded market. The benchmark may also reduce integration costs by identifying agents that require less human oversight.
Expect follow-up research to focus on methods for improving review constraint compliance, such as fine-tuning on review comments or integrating static analysis. Benchmark adoption could drive development of more sophisticated agent scaffolds that explicitly reason about code review norms. If the gap persists, we may see increased emphasis on human-in-the-loop review for AI-generated patches.