Pretraining Data Can Be Poisoned through Computational Propaganda
This paper proposes a method to poison language model pretraining data by injecting malicious content into web crawler data through public discussion interfaces (e.g., forums, comment sections). It also introduces the HalfLife analysis tool to estimate the retention rate of adversarial content in web-crawled data.
This research demonstrates that injecting malicious content through public discussion interfaces can contaminate large-scale pretraining data, and existing data cleaning pipelines are insufficient to fully filter it. The HalfLife tool can be used to assess the practical impact of such attacks.
The HalfLife tool estimates the retention probability of malicious content in pretraining data by analyzing the lifecycle and crawl frequency of web pages. This method provides quantifiable metrics for evaluating data poisoning risks.
This study reveals a new vulnerability in the pretraining data supply chain: third-party web content (e.g., forums, comment sections) can serve as attack vectors. This necessitates the addition of adversarial content detection steps in data collection and cleaning pipelines.
For companies that rely on web-crawled data to train large models, this research highlights potential security risks and may drive demand for data security audit services.
Future developments may include automated detection tools for pretraining data poisoning and more robust data cleaning methods. Additionally, model trainers may need to more rigorously audit data sources, especially user-generated content.