Galactica: A Large Language Model for Science – Reshaping Scientific Knowledge Management
Submitted in November 2022. Galactica is a large language model specifically trained on scientific knowledge, based on massive scientific corpora including papers, references, and knowledge bases. On knowledge probing tasks such as LaTeX equations, it achieves 68.2% accuracy, far surpassing GPT-3's 49.0%; it outperforms Chinchilla and PaLM 540B in mathematical reasoning; it achieves SOTA on PubMedQA and MedMCQA. The model is open-source.
Galactica is the first large language model designed specifically for the scientific domain, capable of storing, combining, and reasoning over scientific knowledge. It significantly outperforms general-purpose large models on scientific tasks such as mathematics and medicine, demonstrating the potential of domain-specific LLMs. Galactica is expected to become a new tool for scientists, aiding literature retrieval, experiment design, knowledge discovery, and even automatic paper generation.
Galactica is based on the Transformer architecture, with parameter count not explicitly stated but estimated around 120B (based on the paper). Training data includes 48 million papers, knowledge bases (e.g., Wiki, StackExchange), textbooks, code, etc., with special handling of scientific formats such as LaTeX, chemical formulas, and protein sequences. It uses causal language modeling objective. On the MMLU math subset, Galactica achieves 41.3% accuracy, outperforming Chinchilla's 35.7%; on the MATH benchmark, it achieves 20.4%, far exceeding PaLM 540B's 8.8%. It achieves 77.6% on PubMedQA and 52.9% on MedMCQA, both SOTA. Additionally, it surpasses BLOOM and OPT-175B on BIG-bench. Limitations include training data cutoff in 2022 and potential inclusion of erroneous information; inference may produce plausible but incorrect scientific content.
Galactica can be applied to automatic summarization of scientific literature, generation of experimental protocols, drug molecule design, and mathematical theorem proving. For industries such as pharmaceuticals, materials science, and academic publishing, it can significantly improve the efficiency of knowledge retrieval and generation. Its open-source nature allows research institutions to customize deployment.
It is recommended that research institutions and pharmaceutical companies deploy Galactica for literature mining and hypothesis generation. A scientific assistant product based on Galactica could be developed to help researchers quickly acquire domain knowledge.
Future focus should be on integrating Galactica into real scientific workflows, such as combining with literature management tools and laboratory record systems. Its scientific reasoning capabilities can be further enhanced through retrieval-augmented generation (RAG). Deployment requires attention to computational resources, but compared to general-purpose models, it offers higher cost-effectiveness for scientific tasks.