Submitted in November 2022. BLOOM is a 176B-parameter open-source decoder-only Transformer language model, trained collaboratively by hundreds of researchers on the ROOTS corpus covering 46 natural languages and 13 programming languages. It achieves competitive performance on various benchmarks, further enhanced by multitask prompt fine-tuning. The model and code are publicly released under a Responsible AI License.
BLOOM is the first fully open-source large-scale language model with broad multilingual coverage, breaking the monopoly of a few tech giants on LLMs. Its training process is transparent, data sources are diverse, and performance is comparable to commercial models like GPT-3. The release of BLOOM has greatly promoted the global AI research community, enabling resource-constrained organizations to conduct research and application development based on LLMs.
BLOOM adopts a standard decoder-only Transformer architecture with 176B parameters, trained on 384 NVIDIA A100 80GB GPUs using 3D parallelism (data, tensor, pipeline). The ROOTS corpus contains 1.5TB of text, covering 46 natural languages (including low-resource languages like Swahili and Urdu) and 13 programming languages. The training process disclosed a detailed carbon footprint report (approximately 50.5 tons CO2eq). On multiple benchmarks, BLOOM performs comparably to GPT-3, with significant advantages on multilingual tasks. Multitask prompt fine-tuning (e.g., T0) further improves zero-shot generalization. Limitations include high inference cost (requiring multiple GPUs) and slightly inferior performance on some English tasks compared to specialized models.
BLOOM's open-source nature allows any organization to deploy LLMs privately, avoiding data leakage and API dependency. It is particularly valuable for multilingual markets (e.g., Europe, Africa, Southeast Asia) and can be directly used for localization translation, multilingual customer service, code generation, etc. Additionally, BLOOM's transparency aids AI safety research.
It is recommended that enterprises with data privacy needs (e.g., finance, healthcare) deploy private instances of BLOOM for internal knowledge management, document generation, etc. BLOOM can be used to develop multilingual industry models, reducing reliance on closed-source models like GPT-4.
Future focus should be on fine-tuned versions of BLOOM (e.g., BLOOMZ) in vertical domains and community-driven continuous improvement. Inference efficiency can be enhanced through quantization, distillation, etc. Deployment requires consideration of hardware costs, but compared to commercial APIs, long-term total cost of ownership may be lower.