Logic Before Language: Pre-pretraining on Formal Derivations Fosters Skill Acquisition and Compressibility
A research paper proposes logic pre-pretraining (Logic-PPT), using formal derivations to initialize language models. In a 100B-token regime, Logic-PPT achieves 80% accuracy on linguistic tasks with 36B fewer tokens than standard initialization and outperforms alternative pre-pretraining baselines.
Researchers introduce logic pre-pretraining (Logic-PPT), a method that initializes language models with formal derivations to impart structural and linguistic biases. Scaling to 100B tokens, Logic-PPT accelerates skill acquisition, reaching 80% accuracy on linguistic tasks using 36B fewer tokens than standard initialization, and surpasses other pre-pretraining approaches.
Formal derivations provide abstract mechanisms—variable binding, quantifier and relational dependencies, predicate-argument composition over long contexts—that are central to natural language. This pre-pretraining induces persistent representational changes, improving sample efficiency and compressibility.
If validated, logic pre-pretraining could reduce the computational cost of training large language models by decreasing the token budget needed for linguistic competence, potentially altering pretraining pipelines in industry.
Reducing the token budget for language model training by 36B tokens could translate to significant cost savings in compute and energy, making advanced language models more accessible and environmentally sustainable.
Next signals include replication on diverse model architectures, scaling beyond 100B tokens, and integration with multimodal or instruction-tuned models. Adoption may depend on the availability of large formal derivation corpora.