AgentHPOBench: A Benchmark For Evaluating LLM Agents as Sequential Hyperparameter Optimizers
AgentHPOBench is a sequential benchmark with 30 executable machine learning tasks across seven research categories. Each task starts with a validated baseline run, after which an agent performs several sequential interventions, observing accumulated configurations, metrics, and logs before proposing the next valid configuration. 12 widely used agents and conventional HPO baselines were evaluated under a unified protocol. Results show current agents exhibit measurable experimental optimization ability across domains but face limitations in sustained iterative refinement, complex log diagnosis, and consistent progress toward reported reference performance.
AgentHPOBench is a new benchmark designed to evaluate LLM agents as sequential hyperparameter optimizers. It comprises 30 executable ML tasks across seven categories, where agents must iteratively propose hyperparameter configurations based on observed experimental evidence. Evaluation of 12 agents and conventional HPO baselines reveals that while agents demonstrate some optimization ability, they struggle with sustained refinement, log diagnosis, and reaching reference performance.
The benchmark moves beyond static code generation to test agents' ability to interpret experimental feedback and adapt hyperparameters sequentially. The observed limitations in iterative refinement and log diagnosis suggest that current LLM agents lack robust mechanisms for long-horizon experimental reasoning and may overfit to immediate metrics without deeper diagnostic understanding.
As LLMs are increasingly positioned as autonomous scientific agents, this benchmark highlights a critical gap in their practical utility for real-world ML experimentation. The results indicate that without improvements in sequential decision-making and log interpretation, agent-based HPO may not yet replace traditional methods in production ML pipelines.
For organizations investing in AI-driven automation of ML workflows, this benchmark provides a standardized way to assess agent capabilities and identify where human oversight remains necessary. It may guide tooling development and procurement decisions for MLOps platforms.
Future work may focus on enhancing agents' memory, planning, and diagnostic capabilities for sustained experimental optimization. Improvements could lead to more reliable autonomous experimentation in scientific research and applied ML, but current limitations suggest a need for hybrid human-agent workflows in the near term.