Fine-Grained Multi Image Object Hallucination Benchmark
A benchmark called MIOH is introduced to systematically evaluate object hallucination in multimodal large language models across four tasks (existence, counting, attribute, position) and three multi-image reasoning patterns (comprehensive, comparative, selective) under three adversarial pressures (visual context scale, perceptual difficulty, contextual bias). Evaluation of 29 models reveals that even state-of-the-art systems like GPT-5 and Gemini-2.5-Pro exhibit distinct failure patterns.
MIOH is a fine-grained multi-image object hallucination benchmark that evaluates object hallucination across four foundational tasks and three multi-image reasoning patterns under controlled adversarial pressures. Evaluation of 29 models shows that even state-of-the-art systems like GPT-5 and Gemini-2.5-Pro exhibit distinct failure patterns across different reasoning patterns and tasks.
The benchmark decomposes hallucination triggers into visual context scale, perceptual difficulty, and contextual bias, enabling diagnosis of how multi-image reasoning demands interact with visual complexity. Distinct failure patterns across reasoning patterns suggest that hallucination is not solely a perceptual issue but also arises from reasoning and attention mechanisms.
The finding that leading models such as GPT-5 and Gemini-2.5-Pro still exhibit object hallucination in multi-image settings indicates a persistent reliability gap for enterprise and consumer applications requiring cross-image reasoning, such as document analysis, surveillance, and visual question answering.
For businesses deploying multimodal AI in multi-image workflows, MIOH provides a diagnostic tool to assess failure modes and select models based on task-specific hallucination rates, potentially reducing errors in high-stakes applications like medical imaging, legal document review, and autonomous systems.
Observable next signals include whether model developers release updates targeting multi-image hallucination, whether MIOH is adopted as a standard evaluation in model cards, and whether downstream applications implement mitigation strategies such as retrieval-augmented verification or explicit reasoning constraints.