Spanish Pre-trained BERT Model and Evaluation Data
This paper releases the first BERT model pre-trained entirely on Spanish data, and integrates multiple Spanish NLP tasks to form a GLUE-like evaluation benchmark. Experiments show that the model outperforms multilingual BERT on most tasks, achieving new state-of-the-art on some. The model, pre-training data, and benchmark are publicly available.
Addressing the scarcity of resources for Spanish, the paper contributes a pure Spanish pre-trained BERT model and a unified evaluation benchmark, significantly improving performance on Spanish NLP tasks, and open-sourcing all resources.
The model adopts the BERT architecture and is pre-trained on large-scale Spanish corpora including Wikipedia, news, and books. The evaluation benchmark covers tasks such as sentiment analysis, named entity recognition, and natural language inference. Compared to multilingual BERT (e.g., mBERT), the Spanish BERT improves accuracy by 2-5 percentage points on most tasks, achieving new SOTA on some (e.g., sentiment analysis). Limitations include not exploring larger models or more advanced architectures.
Spanish is the third most spoken language globally. This model can be directly applied to commercial scenarios such as Spanish customer service, search, and translation, reducing reliance on multilingual models and improving localization effectiveness.
It is recommended that Spanish market teams compare this model with existing mBERT on customer service, content moderation, and search ranking data, evaluating based on accuracy, language variant coverage, and inference cost, without directly extrapolating benchmark gains to production.
Future work needs to validate the model's performance on more downstream tasks (e.g., question answering, summarization) and compare with other Spanish models (e.g., RoBERTa).