How Do VLMs Behave When Blind or Misled? Behavioral Evaluation of VLMs on Scientific Figures
A new arXiv paper introduces SciFigBench, a diagnostic benchmark for vision-language models (VLMs) on scientific figure understanding. It contains 250 figures with human annotations across perception, reasoning, and behavioral reliability under uncertainty, totaling 600+ hours of annotation effort. The benchmark includes over 34,000 evaluation setups via image transformations, reasoning questions, resistance probes, caption-bias probes, and confirmed selective-blur targets. The paper proposes the Admittance-Resistance-Inductance (A-R-I) framework to evaluate whether models acknowledge insufficient evidence, resist misleading context, and infer cautiously from partial information. Results show GPT-5.2 achieves the highest description quality (MQM 91.6) with strong reasoning accuracy (78…).
Researchers introduce SciFigBench, a benchmark for evaluating vision-language models on scientific figures, focusing on behavioral reliability under uncertainty. It includes 250 annotated figures and over 34,000 stress-test setups. The A-R-I framework assesses whether models admit insufficient evidence, resist misleading context, and infer cautiously. GPT-5.2 shows the highest description quality and strong reasoning accuracy.
The A-R-I framework operationalizes behavioral reliability: Admittance (acknowledging insufficient evidence), Resistance (resisting misleading context), and Inductance (cautious inference from partial information). The benchmark's 34,000+ setups, including selective-blur targets and caption-bias probes, enable systematic stress testing of VLM behavior beyond accuracy. Reported MQM 91.6 for GPT-5.2 indicates high description quality, but the paper highlights substantial behavioral differences among models, suggesting that high perception accuracy does not guarantee reliable behavior under uncertainty.
This benchmark addresses a gap in VLM evaluation: most benchmarks measure accuracy, not behavioral reliability. As VLMs are deployed in scientific and high-stakes domains, the ability to admit uncertainty and resist misleading context becomes critical. The introduction of SciFigBench may influence how model developers and enterprises evaluate and select VLMs for scientific figure understanding tasks.
For enterprises using VLMs in scientific or technical workflows, SciFigBench provides a way to assess model reliability under uncertainty, reducing risk of incorrect outputs when visual evidence is missing or misleading. The benchmark's focus on behavioral reliability can inform procurement and deployment decisions, especially in regulated or high-stakes industries.
Expect follow-up work applying the A-R-I framework to other VLM architectures and domains. Model developers may incorporate behavioral reliability metrics into training and evaluation pipelines. The benchmark could become a standard for assessing VLM trustworthiness in scientific applications, potentially leading to improved uncertainty handling in future models.