Pandora's AI Model Routing Box: Efficient Allocation with Costly Value Estimation
A research paper proposes Pandora's Router, a centralized policy for routing queries in heterogeneous AI systems by formalizing the tradeoff between cheap noisy value estimators and expensive accurate ones as an instance of Pandora's Box. It also introduces Pandora's Bidder, a decentralized setting where specialists decide on self-assessment before accepting a price. Experiments cover a multi-LLM benchmark and retrieval-augmented specialists.
The paper addresses efficient query routing in heterogeneous AI systems, where routing decisions require estimating each specialist's expected return. It models the cost of value estimation using the Pandora's Box framework, deriving closed-form value-of-information expressions under a Gaussian signal model. The centralized Pandora's Router and decentralized Pandora's Bidder are evaluated across multiple domains.
The approach leverages optimal search theory to balance estimation cost and routing accuracy. Under Gaussian assumptions, the value of information has closed-form expressions, enabling principled decisions on when to refine estimates. The decentralized variant introduces a market-like mechanism where specialists self-assess before claiming queries.
This research could influence the design of multi-model AI systems and model routing infrastructure, potentially reducing inference costs and improving response quality. The decentralized bidding mechanism may inspire new economic models for AI service marketplaces.
Efficient routing can lower operational costs for AI providers by reducing unnecessary expensive evaluations while maintaining or improving output quality. The decentralized approach may enable new pricing and incentive structures for specialist models.
Observable next signals include adoption of routing policies in production multi-model systems, further empirical validation on diverse benchmarks, and exploration of non-Gaussian signal models or alternative cost structures.