SinkPruner: Sink-Free Visual Token Pruning for Multimodal Large Language Models
SinkPruner is a training-free visual token pruning framework for efficient multimodal large language model (MLLM) inference. It uses a coarse-to-fine design with a visual sanitizer to filter high-norm outlier tokens and alleviate attention sink and attention dispersion, and a text-guided pruner to retain tokens semantically aligned with the text query. Experiments on twelve image-language and four video-language benchmarks demonstrate effectiveness, efficiency, and generalization.
SinkPruner introduces a training-free visual token pruning method for MLLMs that addresses high-norm outlier tokens, which are redundant but often preserved by existing methods. The framework combines a visual sanitizer and a text-guided pruner to reduce computational overhead while maintaining performance across image and video benchmarks.
The method identifies high-norm outlier tokens as redundant in feature and spatial dimensions and removes them to mitigate attention sink and attention dispersion. The coarse-to-fine pruning strategy first sanitizes visual tokens globally, then uses text guidance for semantic alignment, improving inference efficiency without training.
Efficient inference for multimodal models is critical for deployment in resource-constrained environments. Training-free pruning methods like SinkPruner lower the barrier for adopting large MLLMs in production by reducing computational costs without requiring retraining.
Reduces inference costs for multimodal AI services, enabling more scalable and cost-effective deployment of MLLMs in applications such as visual question answering and video understanding.
Further validation on diverse real-world datasets and integration with existing MLLM serving stacks could accelerate adoption. The approach may inspire additional training-free optimization techniques for other model components.