SciDocBench: A Workflow-Centered Benchmark and Data Pipeline for Scientific Document Understanding
SciDocBench is a workflow-centered benchmark for scientific document understanding containing 124 expert-authored questions across 7 capability groups and 19 subtasks in 5 scientific domains. Each question is instantiated under 4 conditions (English/Chinese × all-images-first/interleaved), yielding 496 evaluation instances. The strongest evaluated system achieves 62.6/100, with weaknesses in document perception, evidence grounding, verification, and cross-document reasoning. SciDocIR is introduced as a typed evidence-graph representation preserving document objects, layout, and cross-reference relations.
Researchers introduced SciDocBench, a benchmark designed to evaluate multimodal models on realistic scientific-reading workflows that require joint reasoning over text, equations, figures, tables, code, and datasets. The benchmark includes 124 expert-authored questions across seven research-assistant capability groups and 19 subtasks spanning five scientific domains. Each question is tested under four matched conditions combining English or Chinese questions with all-images-first or interleaved document representations, resulting in 496 evaluation instances. The best-performing system scored only 62.6 out of 100, with notable weaknesses in document perception, evidence grounding, verification, and cross-document reasoning. To convert these diagnostic findings into scalable training signals, the authors also introduced SciDocIR, a typed evidence-graph representation that preserves scientific document objects, layout, and cross-reference relations.
The benchmark's controlled four-condition design isolates the impact of language and document representation on multimodal scientific understanding. The low top score (62.6/100) indicates that current models struggle with integrated reasoning across heterogeneous scientific artifacts. SciDocIR's typed evidence-graph representation suggests a move toward structured, provenance-preserving training data that could improve grounding and verification capabilities.
This work highlights a gap between isolated benchmark performance and real-world scientific assistant workflows. The explicit inclusion of Chinese and English conditions signals attention to multilingual scientific document processing. The introduction of a data pipeline (SciDocIR) points to potential commercial interest in tools for literature review, research assistance, and evidence synthesis.
The benchmark and data pipeline provide a foundation for developing more reliable scientific research assistants, which could serve markets in academic publishing, R&D, and enterprise knowledge management. The multilingual design broadens applicability to non-English scientific communities.
Expect follow-up models trained or fine-tuned on SciDocIR-style evidence graphs to target the identified weaknesses. The benchmark may become a standard evaluation for scientific document understanding, and the data pipeline could be adopted by research assistant products. Watch for improvements in cross-document reasoning and verification as next signals.