ReToken: One Token to Improve Vision-Language Models for Visual Retrieval
ReToken is a single learnable embedding trained as an explicit retrieval target that selects a sparse set of query-relevant visual tokens from a pre-filled visual KV cache. Trained on a small image-QA dataset, it improves Qwen3VL-8B by 13.4 points and InternVL3.5 by 12.4 points on Visual Haystacks, and transfers zero-shot to long video for an 8.0-point gain on LVBench with Qwen3VL-8B. Training and long-video inference fit on a single H100.
ReToken introduces a lightweight, learnable token that acts as a retrieval target to select query-relevant visual tokens from a pre-filled KV cache, addressing performance degradation and memory constraints in vision-language models with long visual contexts. It achieves significant gains on image and video benchmarks, with training and inference possible on a single H100 GPU.
ReToken leverages a pre-filled visual KV cache and a single learnable embedding to perform sparse token selection, avoiding the need to process all visual tokens at once. This approach reduces computational load while maintaining or improving retrieval accuracy, and its zero-shot transfer to long video suggests the learned selection mechanism generalizes across modalities.
The ability to handle long visual contexts efficiently on a single GPU could lower barriers for deploying vision-language models in resource-constrained environments, potentially accelerating adoption in applications like video understanding, surveillance, and content moderation.
By enabling efficient processing of long visual contexts on a single H100, ReToken reduces infrastructure costs and expands the feasibility of deploying advanced vision-language models in edge or cloud scenarios, potentially opening new markets for visual search and video analytics.
If ReToken's approach proves robust across diverse datasets and model architectures, it may become a standard component in vision-language models. Future work could explore its integration with other efficiency techniques and its performance on even longer or more complex visual sequences.