MNIST-PRO: MNIST is Back as a Partially Observable World for AI Agents
MNIST-PRO is a benchmark that isolates agentic perception by converting MNIST digit recognition into a sequential, glimpse-based search task with lookback constraints. Ten multimodal models were evaluated across four memory representations: raw visual history, textual states, structured metric grid maps, and a consolidated visual canvas. Models excel under full observability, but partial observability exposes a clear performance gap. Three bottlenecks were identified: perceptual-state construction and interpretation, premature stopping of exploration, and failure to revise early incorrect beliefs despite contradictory evidence.
The benchmark isolates perceptual-state construction and interpretation by removing physical and control complexities. The four memory representations test different approaches to maintaining an evolving perceptual state under partial observability. The identified bottlenecks suggest that current multimodal models struggle with integrating fragmented glimpses, deciding when to stop exploring, and updating beliefs in light of new evidence.
This research highlights a gap in agentic perception capabilities for multimodal models, which could affect applications requiring active sensing and working memory coordination, such as robotics, autonomous systems, and interactive AI. The benchmark provides a standardized way to measure progress in this area.
The benchmark offers a tool for evaluating and comparing multimodal models on agentic perception tasks, which could inform model selection and development for applications in robotics, autonomous systems, and interactive AI.
Future work may focus on improving perceptual-state construction, exploration strategies, and belief revision in multimodal agents. The benchmark could be extended to more complex environments or used to track progress in agentic perception over time.