VAKRA: Evaluating Multi-Hop Reasoning Across APIs and Retrieval Under Tool-Use Policies
VAKRA is a benchmark of over 8,000 executable APIs across 62 domains with tasks spanning three settings: diverse API interaction styles, multi-hop reasoning over structured APIs, and multi-source reasoning with natural-language tool-use policy constraints. Correctness is verified by re-executing predicted tool calls against live APIs. Using a fixed ReAct harness, the best model achieves 70.4% on single-hop endpoint-style tasks and drops to 50-51% on compositional APIs; performance degrades by over 50% as reasoning depth increases, and policy-constrained questions expose failures as low as 2.4% on unanswerable queries.
The benchmark isolates model capabilities from agent architecture by using a fixed ReAct harness. Trace analysis shows failures concentrate at language-mediated reasoning, including entity disambiguation and other reasoning steps. The use of live API re-execution for correctness verification allows multiple valid paths, making the evaluation more robust than static answer matching.
Enterprise agents must combine structured API calls with document retrieval under policy constraints. Current models show significant degradation on compositional and policy-constrained tasks, indicating a gap between single-hop tool use and real-world multi-hop enterprise workflows. This benchmark provides a standardized way to measure progress in this area.
For enterprises deploying AI agents, VAKRA offers a way to evaluate model suitability for multi-hop API and retrieval tasks under policy constraints. The low performance on unanswerable queries (2.4%) highlights a risk in production systems where agents may hallucinate or take incorrect actions when faced with policy restrictions.
Observable next signals include whether frontier models improve on VAKRA's compositional and policy-constrained tasks in subsequent releases, and whether agent architectures beyond ReAct (e.g., planning-based or retrieval-augmented) close the performance gap. Adoption of VAKRA by model developers or enterprise evaluation suites would indicate industry recognition.