Bandits in Prod: Hyperparameter Optimization at Inference Time
A paper titled 'Bandits in Prod: Hyperparameter Optimization at Inference Time' was published on arXiv on 2026-09-01. It formalizes Online Hyperparameter Optimization (OHPO) as an infinitely many-armed bandit over mixed and conditional search spaces, and introduces IMABO, a framework combining any bandit policy with any oracle for proposing new configurations. The paper also introduces IMOSS, a restart-free anytime policy with active set growing as t^β, and proves an expected cumulative quantile-regret bound of O(p_ρ^{-1/β} + T^{(1+β)/2}).
The paper addresses the problem of tuning hyperparameters at inference time in production systems, where configurations can only be evaluated through live requests with noisy feedback. It proposes IMABO, a general framework that combines bandit policies for selecting among sampled configurations with oracles for proposing new ones. IMOSS is introduced as a restart-free anytime policy with a proven regret bound. The paper also discusses three practical oracles: a Tree-structured Parzen Estimator, an incumbent-mutation oracle driven by a per-coordinate bandit, and a pretrained tabular foundation model.
The IMOSS policy's active set grows as t^β, where β ∈ (0,1) controls the trade-off between exploration and exploitation. The regret bound O(p_ρ^{-1/β} + T^{(1+β)/2}) suggests that the choice of β significantly impacts performance, with smaller β leading to slower active set growth but potentially better regret in terms of T. The use of a pretrained tabular foundation model as an oracle indicates a trend toward leveraging pre-trained models for hyperparameter proposal.
This research is directly relevant to production AI systems, especially agentic systems that make inference-time choices such as model selection, retrieval depth, prompting strategy, and decoding temperature. The ability to optimize these choices online without representative validation data could reduce operational overhead and improve system performance. The framework's generality suggests potential adoption in various AI service platforms.
The framework could enable AI service providers to automatically tune inference-time parameters, potentially improving response quality, reducing costs, and increasing user satisfaction without manual intervention. This may lead to more efficient resource utilization and competitive advantages for platforms that implement such optimization.
Observable next signals include follow-up papers applying IMABO to specific agentic systems, open-source implementations of IMOSS, and industry adoption in model serving platforms. The use of pretrained tabular foundation models as oracles may lead to further research on transfer learning for hyperparameter optimization.