Information Abundance Paradox: Long-Context Training Undermines Parametric Knowledge
A study on arXiv (2608.12218v1) proposes the Information Abundance Paradox, hypothesizing that abundant relevant information in training context reduces parametric encoding and increases context reliance. In pretraining with long documents, increasing context window improves language modeling, natural language understanding, and closed-book MCQA only up to an intermediate optimum, after which performance declines. In supervised fine-tuning, more task-relevant train-time context improves performance with supporting context but reduces robustness when context is absent or misleading at test time.
Large language models are increasingly trained with long contexts, but a new arXiv paper challenges the assumption that longer contexts always help. The authors propose the Information Abundance Paradox: when training context contains abundant relevant information, models may rely on context instead of encoding knowledge parametrically. Experiments show that increasing context window in pretraining improves performance only up to an intermediate optimum, then declines. In supervised fine-tuning, more train-time context helps with supporting context but hurts robustness when context is absent or misleading.
The paper suggests that longer context provides a lower complexity solution, shifting learning from parametric internalization to contextualization. This implies a trade-off between context length and parametric knowledge retention, with potential implications for model architecture and training data curation.
The findings may influence how AI labs design training pipelines, balancing context length against parametric knowledge robustness. Products relying on long-context models may need to account for reduced closed-book performance.
Understanding this paradox can help AI companies avoid over-investing in long-context training that degrades parametric knowledge, potentially saving compute costs and improving model reliability in context-free scenarios.
Future research may explore optimal context lengths for different tasks and methods to mitigate the paradox, such as curriculum learning or explicit parametric knowledge objectives. Industry may see a shift toward hybrid approaches that combine long context with strong parametric priors.