The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only
This paper demonstrates that properly filtered and deduplicated web data (solely from CommonCrawl) is sufficient to train powerful large language models, even outperforming models trained on curated corpora like The Pile. The research team extracted 5 trillion tokens from CommonCrawl and released the RefinedWeb dataset of 600 billion tokens, along with 1.3B and 7.5B parameter language models trained on this dataset.
The paper challenges the conventional view that large language models must rely on curated high-quality corpora (e.g., books, papers) to achieve good performance. Through systematic filtering and deduplication, using only web crawl data can train models with superior performance, and high-quality web data remains abundant, with no scarcity issue.
The paper proposes a complete web data cleaning pipeline, including language filtering, quality filtering (based on heuristic rules and classifiers), and exact deduplication (MinHash and fuzzy deduplication). Applied to 5 trillion tokens of CommonCrawl data, the pipeline yields the RefinedWeb dataset. Falcon models with 1.3B and 7.5B parameters were trained and outperformed models trained on The Pile across multiple zero-shot tasks. Evaluation covers language modeling, question answering, reasoning, etc., but does not involve multimodal or code generation. Limitations include data only from English CommonCrawl and relatively small model sizes.
This research has significant implications for the AI industry, showing that high-quality web data is sufficient to support large model training, reducing the industry's reliance on expensive curated datasets. This may change data procurement strategies, encourage more companies to adopt web data cleaning solutions, and accelerate the construction of open-source datasets.
It is recommended that data engineering teams evaluate the RefinedWeb data cleaning pipeline and consider integrating it into internal data pipelines. For model training, the public 600B RefinedWeb subset can be prioritized for experiments to reduce data procurement costs.
Future validation signals include: performance of larger models (e.g., 40B+) on RefinedWeb; multilingual extension; and comparison with other data mixing strategies.