Efficient SWE Agent Benchmarking via Trajectory-Aware Evaluation
PTA-IRT, a Privileged Trajectory-Aware Item Response Theory framework, fuses process and outcome signals from historical execution trajectories to improve calibration subset selection and ability estimation for software engineering agent benchmarks. Under low calibration budgets, PTA-IRT outperforms prior IRT baselines on score and ranking recovery across four SWE benchmarks. Code and data are publicly available at https://github.com/DeepSoftwareAnalytics/PTA-IRT.
Evaluating software engineering agents on realistic benchmarks is costly because each task may require multi-step code exploration, modification, and test execution. Existing efficient evaluation methods select representative subsets to estimate full-benchmark performance but are largely result-only, discarding how agents solve problems. PTA-IRT uses historical execution trajectories as privileged information to provide process-level evidence beyond pass/fail, such as explored context, attempted edits, and solving paths. Under low calibration budgets, PTA-IRT consistently outperforms prior IRT baselines on score and ranking recovery across four SWE benchmarks.
PTA-IRT extends item response theory by incorporating privileged trajectory information, enabling more accurate ability estimation from limited calibration data. The framework leverages process-level signals like explored context and attempted edits, which are typically discarded in result-only evaluation. This suggests that trajectory data can serve as a strong prior for efficient benchmarking, potentially reducing the number of full benchmark runs needed to evaluate new agents.
Efficient evaluation methods are critical for reducing the cost and time required to benchmark software engineering agents, which is a growing need as agentic coding tools proliferate. PTA-IRT's approach could lower the barrier for organizations to evaluate and compare agents on large benchmarks, enabling faster iteration and more informed adoption decisions.
PTA-IRT can reduce computational and financial costs associated with benchmarking software engineering agents, making it more feasible for companies to evaluate multiple agents or frequent model updates. The public code and data enable immediate adoption and integration into existing evaluation pipelines.
If trajectory-aware evaluation gains adoption, we may see more benchmarks and evaluation platforms incorporate process-level signals. This could lead to standardized efficient evaluation protocols and possibly new metrics that capture solution quality beyond pass/fail. Further research may explore applying similar privileged information techniques to other agent domains.