From Terminology to Diagrams: Visual-Instruction Generation for Scientific Diagram Understanding
A framework generates diagram-grounded instruction data using terminology from scientific curricula. It extracts domain concepts, synthesizes atomic facts, retrieves diagrams from the web, and generates captions and multiple-choice questions. The resulting SciGram dataset contains over 194K diagrams and 1.4M visual instructions across life, earth, and physical sciences. Models fine-tuned on SciGram achieve substantial improvements on diagram-centric benchmarks including TQA, ScienceQA, and AI2D, outperforming or matching state-of-the-art VLMs while using fewer training instances. Augmenting LLaVA OneVision with SciGram establishes new state-of-the-art performance on diagram understanding.
Researchers introduced a pipeline that leverages scientific terminology to create large-scale diagram-grounded instruction data. The pipeline extracts domain concepts, synthesizes atomic facts, retrieves relevant diagrams from the web, and generates multimodal supervision (captions and multiple-choice questions). The resulting SciGram dataset comprises over 194K diagrams and 1.4M visual instructions across life, earth, and physical sciences. Fine-tuning on SciGram yields substantial improvements on diagram-centric benchmarks (TQA, ScienceQA, AI2D), outperforming or matching state-of-the-art VLMs with fewer training instances. Augmenting LLaVA OneVision with SciGram sets new state-of-the-art performance on diagram understanding.
The approach uses noisy web data and synthetic annotations, yet fine-tuned models achieve strong benchmark results, suggesting that large-scale, curriculum-derived visual instruction data can effectively teach diagram semantics. The framework's terminology-driven fact synthesis and diagram retrieval may generalize to other structured visual domains. Future work could explore scaling to more scientific disciplines or improving diagram retrieval quality.
Improved scientific diagram understanding could enhance educational technology, automated assessment, and scientific document analysis. The dataset and method may lower the barrier for domain-specific VLM fine-tuning, potentially accelerating adoption in science education and research tools.
The framework enables cost-effective creation of domain-specific visual instruction data, reducing reliance on expensive human annotation. This could lead to more accurate AI tutors, automated grading systems, and scientific literature analysis tools, creating opportunities in edtech and research software.
Expect further research on diagram-grounded instruction generation, possibly extending to other visual modalities (charts, schematics) and languages. Commercial applications may emerge in e-learning platforms and scientific search. Watch for open-source releases of SciGram and fine-tuned models.