A Universal Context-Reuse Layer for Cross-Model KV Sharing
A study on cross-model KV sharing translates KV states from a source model to a target model, including models differing in scale, architecture, attention configuration, tokenizer, and model family. For Qwen2.5-7B to Qwen2.5-1.5B, translated KV states improve LongBench2 accuracy from 27.59% to 34.48% over the native 1.5B baseline. For Qwen2.5-1.5B to Gemma-2-2B, KV handoff reduces target-side prefill cost by up to 67.05% at 4K context length while maintaining decoding perplexity close to native.
Researchers propose a universal context-reuse layer that enables cross-model KV sharing, allowing a target model to reuse KV states produced by a different source model. The approach is evaluated in within-family (Qwen2.5-7B to Qwen2.5-1.5B) and cross-family (Qwen2.5-1.5B to Gemma-2-2B) settings. In the within-family case, translated KV states improve LongBench2 accuracy by 6.89 percentage points over the native 1.5B baseline. In the cross-family case, KV handoff reduces target-side prefill cost by up to 67.05% at 4K context length while maintaining decoding perplexity close to native.
The method translates KV states across models with different architectures and tokenizers, suggesting a learned mapping or projection between KV spaces. The reported accuracy gain in the within-family setting indicates that the translated KV states carry useful contextual information that the smaller target model can exploit. The prefill cost reduction in the cross-family setting implies that the translation overhead is lower than native prefill, making the approach computationally beneficial.
Cross-model KV sharing could reduce redundant prefill computation in multi-model serving environments, such as model routing or cascades, where multiple models process the same context. This may lower inference costs and improve latency for applications that use heterogeneous model fleets.
The approach could enable more efficient use of compute in multi-model LLM serving, potentially reducing infrastructure costs for providers that run multiple models. It may also facilitate model cascades where a smaller model benefits from context processed by a larger model.
Next signals include publication of full experimental details, open-source release of the translation layer, and evaluation on larger model families or production serving stacks. Adoption may depend on the generality of the translation mechanism and its robustness across diverse architectures.