ExecCritic: Learn to Test, Test to Improve for Coding Agents
ExecCritic combines a test-verify-revise scaffold with role-specific reinforcement learning for coding agents. A Test agent independently generates repository-native tests, a fail-closed harness qualifies and freezes them, and a Repair agent revises source code from execution feedback without changing tests. Both roles use Qwen-3.5-35B-A3B as the backbone and are trained separately. On SWE-bench Verified, test quality determines whether feedback helps.
ExecCritic introduces a scaffold that separates test construction from source-code repair to improve coding agents. The Test agent learns to produce behaviorally valid tests that distinguish correct from incorrect patches, while the Repair agent learns direct task resolution and feedback-guided revision. The approach addresses the problem of agent-generated tests encoding incorrect behavioral targets, which can create false confidence when the same trajectory writes both patch and test.
The key technical contribution is the separation of test generation and repair into distinct roles with independent reinforcement learning, using a fail-closed harness to freeze tests before repair. This prevents test-patch error agreement and ensures execution feedback reflects true behavioral requirements. The use of Qwen-3.5-35B-A3B as a shared backbone suggests the method is applicable to mid-size open models.
ExecCritic addresses a critical reliability gap in autonomous coding agents: self-generated tests can validate incorrect fixes. By decoupling test and repair roles, the approach may improve trust in AI-generated code for enterprise software development, where verification quality directly impacts deployment safety.
Improved test quality and repair reliability could reduce the cost of human code review and increase the autonomy of AI coding agents in production environments. Enterprises may benefit from fewer false-positive fixes and more dependable automated software maintenance.
Observable next signals include whether ExecCritic's test-verify-revise scaffold is adopted in other coding agent frameworks, whether the role-specific RL recipe transfers to larger or different base models, and whether the approach generalizes beyond SWE-bench Verified to real-world repositories with less standardized test infrastructure.