AI4AI-Bench: Benchmarking LLM Agents in Algorithmic Design for Recursive Self-Improvement
AI4AI-Bench is a benchmark of 10 frozen research repositories spanning 10 training algorithm families. In each task, an agent has 4 hours on one B300 to rewrite the training algorithm; its code is then rerun from scratch for up to 12 hours and scored by a fixed evaluator hidden from the agent, against the repository's original algorithm under the same procedure. Every task is mapped onto one scale where 0 is an uninformative model, 0.1 is the algorithm the repository ships, and 1.0 is the target.
Recursive self-improvement (RSI) asks whether an AI system can improve the process that produces AI systems. The process is the training algorithm. AI4AI-Bench isolates the ability of an agent to design training algorithms by providing 10 frozen research repositories and scoring rewritten algorithms against original ones under identical procedures.
The benchmark uses a fixed hidden evaluator and a normalized scale (0 uninformative, 0.1 shipped algorithm, 1.0 target) to compare incommensurable metrics across 10 training algorithm families. Agents are constrained to 4 hours on one B300 for rewriting and up to 12 hours for rerun, focusing on algorithmic changes rather than data collection or hyperparameter tuning.
This benchmark addresses a gap in evaluating LLM agents for algorithmic design, which is critical for recursive self-improvement. It may influence how AI labs assess and develop agents capable of improving training processes, potentially shifting focus from data-centric to algorithm-centric improvements.
The benchmark could help organizations evaluate and procure AI systems with algorithmic design capabilities, potentially reducing training costs and improving model efficiency through better training algorithms.
Observable next signals include adoption of AI4AI-Bench in AI research communities, publication of agent performance results on the benchmark, and development of similar benchmarks for other aspects of the AI development pipeline.