Probing Factual Knowledge Transfer with Training Data Interventions
A study proposes an intervention-based framework to test whether multilingual language models transfer factual knowledge across languages during continued pretraining. Starting from an English-pretrained model, researchers continued pretraining on Persian data with specific facts removed at varying granularity. They constructed SIFT, a resource of 500 triples across 20 relations, stratified by cultural origin of the fact's subject into general and Persian-related entities, with natively written Persian cloze templates. Results show fact transfer is very limited: under the strictest removal condition, a large majority of English-acquired facts fail to transfer into Persian. Sentence-level co-occurrence removal is insufficient to eliminate fact signal. Easier (randomly selected) negative candidate sets substantially inflate apparent transfer by rewarding shallow associative heuristics, while performance on a harder candidate set is lower.
Researchers investigated whether multilingual language models transfer factual knowledge across languages during continued pretraining. They continued pretraining an English-pretrained model on Persian data with specific facts removed at varying granularity, using a new resource called SIFT (500 triples, 20 relations, stratified by cultural origin, with Persian cloze templates). The study found that fact transfer is very limited: under the strictest removal condition, most English-acquired facts fail to transfer into Persian. Sentence-level co-occurrence removal was insufficient to eliminate fact signal. Easier negative candidate sets inflated apparent transfer by rewarding shallow associative heuristics, while performance on harder candidate sets was lower.
The intervention-based framework isolates the contribution of target-language training data to factual knowledge. The finding that sentence-level co-occurrence removal is insufficient suggests facts are encoded redundantly or at sub-sentence granularity. The discrepancy between easy and hard negative candidate sets indicates that models may rely on shallow associative heuristics rather than robust factual recall, and evaluation design significantly affects measured transfer.
This research implies that multilingual models may not automatically transfer factual knowledge across languages; continued pretraining on a new language does not guarantee the model knows facts in that language unless they appear in the target-language data. For practitioners building multilingual systems, this suggests the need for explicit data curation and evaluation in each target language, especially for culturally specific facts.
For companies deploying multilingual AI, this research highlights the risk of assuming knowledge transfers across languages. It may drive investment in language-specific data collection and evaluation, and influence the design of multilingual training pipelines to ensure factual accuracy in all supported languages.
Next signals include follow-up studies testing transfer in other language pairs, investigations into what granularity of data removal is necessary to eliminate fact signal, and development of better evaluation sets with harder negative candidates to avoid inflated transfer estimates. The SIFT resource may be used for benchmarking multilingual factual knowledge.