OctoLong: Mid-Training On Cross-Repository Code Contexts Enhances Long-Context Modeling
Researchers introduced OctoLong, a context engineering pipeline that uses an AST parser, language server backend, and package manager to recursively retrieve code references, creating dependency-rich code contexts of millions of tokens. They trained OctoLong-Instruct, a suite of open long-context language models from 600M to 14B parameters, via context-extension mid-training on a ~50B-token mixture containing ~6.2B tokens of OctoLong code contexts, followed by ~10B tokens of instruction tuning. Training ablations and evaluations against 18 state-of-the-art open-weight long-context LMs showed that replacing just 12% of traditional context-extension corpora with OctoLong data yields substantial gains in long-range retrieval.
OctoLong is a pipeline for generating long-context code data by recursively resolving cross-repository dependencies. Mid-training on this data significantly improves long-range retrieval in language models, outperforming 18 existing open-weight long-context models when only 12% of the training mixture is OctoLong data.
The key innovation is using program analysis tools (AST parser, language server, package manager) to automatically construct dependency graphs across code repositories, producing training contexts with genuine long-distance dependencies. This contrasts with existing long-context corpora that often lack such structure. The mid-training recipe demonstrates that a small fraction of high-quality, dependency-rich data can substantially boost long-context performance.
This work suggests that synthetic data generation pipelines leveraging software engineering tools can create valuable training data for long-context models. It may influence how organizations curate pre-training and fine-tuning data for code-focused AI assistants, potentially reducing reliance on scarce natural long-context sources.
Improved long-context code understanding can enhance developer productivity tools, enabling more accurate code generation, bug detection, and refactoring across large codebases. This could lead to more capable enterprise AI coding assistants and reduce the cost of maintaining complex software systems.
Next signals include adoption of similar dependency-aware data pipelines in other domains (e.g., legal, scientific literature), scaling the approach to larger models, and integration into commercial code assistants. Further research may explore whether such mid-training improves agentic task performance in software development workflows.