Algebraic Decomposition Theory for Transformer Length Generalization
A paper establishes the first complete characterization of which regular languages transformers length-generalize on and provides a decision algorithm running in polynomial time in the size of the language's syntactic monoid. The results rely on an effective characterization of the regular languages in C-RASP. Classical tools like Krohn-Rhodes decomposition theory are insufficient for C-RASP because flip-flop and simple groups are not expressible in C-RASP, and unbounded counting is not expressible by finite semigroups.
Transformer-based language models sometimes generalize to sequences longer than seen during training, but a precise characterization of which tasks admit length generalization was lacking. This work provides the first complete characterization of which regular languages transformers length-generalize on, along with a polynomial-time decision algorithm based on the syntactic monoid. The characterization uses C-RASP, a formalism expressing length-generalizable languages, and shows that classical Krohn-Rhodes decomposition theory is insufficient because its basic building blocks are not expressible in C-RASP, while C-RASP's unbounded counting is not expressible by finite semigroups.
The paper bridges formal language theory and transformer capabilities by characterizing length generalization through C-RASP. It demonstrates that Krohn-Rhodes decomposition theory cannot be directly applied because flip-flop and simple groups are not expressible in C-RASP, and unbounded counting in C-RASP is not expressible by finite semigroups. This suggests a new algebraic decomposition theory tailored to transformer length generalization, potentially involving semigroups with counting features.
This theoretical result could guide the design of architectures and training methods that reliably generalize to longer sequences, which is critical for deployment in real-world applications where input lengths vary. It may also inform benchmarks and evaluation protocols for length generalization, helping practitioners identify which tasks are feasible for current transformer models.
For enterprises using transformers on variable-length data (e.g., code, logs, DNA), this characterization can help predict when models will fail on longer inputs and guide data augmentation or architecture choices. It may reduce risk in deploying models to production where length generalization is assumed but not guaranteed.
Next signals include follow-up work extending the characterization to context-free or other language classes, empirical validation of the decision algorithm on practical regular languages, and potential integration of C-RASP-based analysis into model interpretability tools. Watch for papers applying these algebraic methods to improve transformer length generalization in practice.