MM-ToolSandBox: A Unified Framework for Evaluating Visual Tool-Calling Agents
MM-ToolSandBox is a benchmark and evaluation framework for visually grounded tool-calling agents. It provides a stateful execution environment spanning 500+ tools across 16 application domains, supporting multi-image, multi-turn tasks. An automated scenario generation pipeline produces 258 human-verified nominal scenarios and 50 variants. Evaluating 12 state-of-the-art models shows the best model achieves below 50% success rate. Failure analysis reveals 53% of failures stem from incorrect information extraction from images.
The benchmark reveals that visual precision is a primary bottleneck for capable models, with 53% of failures due to incorrect information extraction from images despite correct task workflows. This suggests that improving visual grounding accuracy is a critical next step for tool-calling agents.
The benchmark's coverage of 500+ tools across 16 application domains indicates a growing need for standardized evaluation of multimodal agent capabilities. The low success rates (below 50%) suggest that current models are not yet ready for reliable deployment in visually grounded tool-use scenarios.
The benchmark provides a standardized way to evaluate visual tool-calling agents, which is valuable for companies developing such agents. The low performance of current models highlights a market opportunity for improved visual grounding solutions.
Future work should focus on improving visual information extraction accuracy, as indicated by the failure analysis. The benchmark's automated scenario generation pipeline could be extended to cover more domains and interactive scenarios.