Structural Silence: When AI Infrastructure Fails Speakers of Underrepresented Languages
A paper on arXiv (cs.AI) examines structural barriers in AI infrastructure for underrepresented languages, using Bengali as a case study. It reports Bengali accounts for less than 0.5% of global web content despite representing nearly 4% of the global population; a 67:1 training-token deficit between English and Bengali in major multilingual corpora; a tokenization penalty from Bengali's alphasyllabary script; and rural internet penetration of 36.5% versus 71.4% urban.
The paper identifies four interlocking failures disadvantaging Bengali speakers in AI-assisted education: severe web presence gap, training-token deficit, tokenization penalty, and connectivity exclusion. These reflect longstanding resource-allocation decisions in AI infrastructure.
The tokenization penalty for Bengali's alphasyllabary script compounds the data deficit through higher token fertility, meaning models require more tokens per unit of text, increasing cost and reducing efficiency. This is a measurable infrastructure-level bias that persists even with equal corpus size.
AI tools for education and language support are framed as scalable, but underlying infrastructure choices systematically exclude underrepresented languages. This creates a market failure where demand exists but supply is structurally constrained, limiting adoption in low-connectivity regions.
Addressing these barriers could unlock AI-assisted education for nearly 4% of the global population, representing a significant underserved market. Reducing token fertility and improving connectivity could lower costs and expand reach.
Observable next signals include: new multilingual corpora with improved Bengali representation, tokenizer updates for alphasyllabary scripts, and deployment architectures optimized for low-connectivity environments. Watch for benchmarks measuring token fertility across scripts.