BioMedLM: A 2.7B Parameter Language Model Trained On Biomedical Text
Submitted in March 2024. BioMedLM is a 2.7B parameter GPT-style autoregressive model trained only on PubMed abstracts and full texts. After fine-tuning, it achieves 57.3% on MedMCQA and 69.0% on MMLU Medical Genetics, competing with larger models. The model is open-sourced, aiming to provide a transparent, privacy-preserving, cost-effective and environmentally friendly foundation for biomedical NLP.
BioMedLM demonstrates that domain-specific small language models can compete with large models (e.g., GPT-4) on key benchmarks while offering lower computational cost, better privacy protection, and interpretability. Its success hinges on high-quality domain data pre-training and targeted fine-tuning. This provides a viable path for AI applications in specialized fields like healthcare and law, challenging the prevailing 'bigger is better' notion.
The model is based on the GPT-2 architecture with 2.7B parameters, pre-trained on PubMed abstracts and full texts (approximately 20 million articles). Fine-tuning uses standard instruction fine-tuning, achieving strong results on MedMCQA and MMLU medical subsets. Compared to GPT-4 and Med-PaLM 2, BioMedLM has two orders of magnitude fewer parameters, yet the performance gap is acceptable. The model supports local deployment, avoiding data leakage, and has lower training and inference energy consumption.
This model has direct value for the healthcare AI industry: hospitals and pharmaceutical companies can deploy it locally for clinical decision support, literature retrieval, patient Q&A, etc., without worrying about data privacy. Additionally, its open-source nature lowers industry barriers, fostering innovation in biomedical NLP applications. Moreover, this paradigm can be extended to other specialized fields such as law and finance.
It is recommended that healthcare IT companies evaluate BioMedLM as a privately deployed medical NLP engine for building products such as intelligent diagnosis and medical record analysis. Investment institutions should pay attention to startups based on domain-specific small models, especially in industries with high compliance requirements.
Future work should focus on the model's performance on a broader range of biomedical tasks (e.g., relation extraction, text generation), as well as the feasibility of continued pre-training and incremental updates. Furthermore, combining with retrieval-augmented generation (RAG) may further enhance practicality.