ConvergeFlow: Language Flow with Provable Convergence to Token Embeddings
ConvergeFlow is an embedding-space flow-based language model that constrains the data predictor to the convex hull of token embeddings and trains solely with mean squared error. Under regularity conditions, the flow provably converges to valid token embeddings, enabling direct token prediction without a cross-entropy-supervised decoder. Three sampling mechanisms control the trade-off between generative perplexity and entropy. Experiments on OpenWebText show performance competitive with existing continuous and discrete diffusion LMs. Code is available at https://github.com/Na-Li66…
ConvergeFlow introduces a flow-based language model in embedding space with provable convergence to token embeddings, removing the need for a cross-entropy decoder. It achieves competitive performance on OpenWebText and offers sampling mechanisms to balance perplexity and entropy.
The key technical contribution is constraining the data predictor to the convex hull of token embeddings and using flow matching with MSE loss, which guarantees convergence to valid token embeddings under regularity conditions. This eliminates the cross-entropy decoder typically required in continuous diffusion LMs. Three sampling mechanisms are developed to adjust the generative perplexity-entropy trade-off.
This work suggests a shift toward decoder-free continuous language models, potentially simplifying training pipelines and reducing reliance on discrete token prediction. If the approach scales, it could influence model architectures in both research and commercial settings.
The method could reduce training complexity and computational cost by removing the cross-entropy decoder, and the sampling mechanisms offer controllable diversity for generation tasks. This may be valuable for companies developing efficient language models.
Next signals to watch include scaling ConvergeFlow to larger datasets and model sizes, comparisons with autoregressive LMs on standard benchmarks, and adoption or replication by other research groups. The release of code may accelerate follow-up work.