Beyond Final Scores: A Systematic Evaluation of Agents for Long-Horizon AI Research and Development
A systematic evaluation of seven frontier models on 36 long-horizon tasks was conducted using a new framework with rule-based metrics to characterize within-run behavior through Solution Framing, Execution, and Feedback Control, and controlled comparisons to assess experience reuse within and across tasks. Results show current agents operate more like engineering optimizers than fully autonomous researchers: they can formulate and implement practical solutions, but performance varies substantially across runs, strongest solutions mainly adapt or combine established techniques, and genuine methodological novelty remains rare. Detailed analysis reveals distinct process bottlenecks behind similar final outcomes.
The paper presents a systematic evaluation of seven frontier models on 36 long-horizon tasks, using a new framework that goes beyond final scores to characterize within-run behavior through Solution Framing, Execution, and Feedback Control, and to assess experience reuse within and across tasks. The results indicate that current agents function more like engineering optimizers than fully autonomous researchers: they can formulate and implement practical solutions, but their performance varies substantially across runs, their strongest solutions mainly adapt or combine established techniques, and genuine methodological novelty remains rare. The analysis also reveals that similar final outcomes can arise from distinct process bottlenecks.
The evaluation framework uses rule-based metrics to decompose agent behavior into Solution Framing, Execution, and Feedback Control, enabling identification of process bottlenecks that are not visible from final scores alone. Controlled comparisons assess experience reuse within and across tasks, providing a more granular view of agent learning and adaptation. The finding that agents primarily adapt or combine established techniques suggests current models lack the capacity for genuine methodological novelty in long-horizon research tasks.
This research highlights a gap between the perception of autonomous AI researchers and the current reality: agents are better characterized as engineering optimizers. For organizations investing in AI-driven R&D, this implies that while agents can assist with practical solution formulation and implementation, they are not yet reliable for generating novel research directions. The variability in performance across runs also suggests that deployment of such agents requires robust oversight and multiple trials to ensure consistent outcomes.
For enterprises and research labs, this work provides a more rigorous way to evaluate AI agents for long-horizon tasks, potentially reducing the risk of overestimating agent capabilities. The framework can be used to benchmark and select agents for specific R&D workflows, and the insights into process bottlenecks can inform targeted improvements. However, the finding that agents mainly adapt existing techniques suggests that near-term business value lies in using agents for well-defined engineering optimization rather than open-ended research.
Observable next signals include: (1) improvements in agent architectures that explicitly target methodological novelty, such as enhanced exploration or hypothesis generation modules; (2) development of more sophisticated evaluation benchmarks that measure process quality and experience reuse, not just final task success; (3) increased reporting of within-run behavior and bottleneck analysis in agent research papers; and (4) potential integration of these evaluation frameworks into agent development pipelines to guide iterative improvement.