The Falcon Series of Open Language Models: Falcon series open-source language models, up to 180B parameters, performance close to GPT-4
The Falcon series includes causal decoder models with 7B, 40B, and 180B parameters, trained on over 3.5 trillion tokens of high-quality web data, representing the largest publicly recorded pre-training run. Falcon-180B significantly outperforms PaLM, Chinchilla, LLaMA 2, and Inflection-1, with performance close to PaLM-2-Large, while having lower pre-training and inference costs, making it one of the world's top three language models alongside GPT-4 and PaLM-2-Large.
The Falcon series achieves performance close to GPT-4 through large-scale pre-training on high-quality web data and efficient distributed training tools, while reducing costs and open-sourcing models and datasets to promote open science.
Falcon adopts a causal decoder architecture, efficiently trained on AWS cloud infrastructure using 4096 A100 GPUs with a custom distributed training codebase. Evaluations show Falcon-180B surpasses PaLM, Chinchilla, and LLaMA 2 on multiple benchmarks, approaching PaLM-2-Large. The model and a 600B token web dataset are open-sourced under a permissive license.
The open-source release of Falcon-180B lowers the barrier for top-tier language models, potentially driving enterprise applications and academic research, challenging the market position of closed-source models like GPT-4.
Enterprises can consider adopting Falcon-180B for private deployment, leveraging its open-source nature to reduce licensing costs, and customizing it through fine-tuning to meet specific business needs.
Future attention should be paid to community validation of Falcon-180B on more downstream tasks, expansion of the open-source ecosystem, and support for multimodal and instruction tuning in subsequent versions.