LocalMamba: Visual State Space Model with Windowed Selective Scan
Submitted in March 2024. LocalMamba proposes a local scanning strategy that divides images into windows to capture local dependencies and dynamically searches for the optimal scanning pattern per layer. On ImageNet, LocalMamba achieves 3.1% higher accuracy than Vim-Ti with the same 1.5G FLOPs. Code is open-sourced.
This work addresses the scanning direction optimization problem in visual Mamba (ViM) by proposing windowed local scanning and layer-wise dynamic search, significantly improving the performance of visual state space models. The key insight is that preserving local 2D dependencies is crucial for vision tasks, and different layers have different preferences for scanning patterns. This provides a critical improvement for Mamba's application in vision, surpassing CNN and ViT of comparable scale.
Traditional ViM flattens images into 1D sequences, destroying local spatial structure. LocalMamba introduces window scanning, scanning in raster order within each window and connecting windows sequentially, thus maintaining local continuity. Additionally, a differentiable search method selects the optimal scanning direction (e.g., horizontal, vertical, diagonal) independently for each layer, with a search space containing multiple patterns. Experiments on ImageNet classification, COCO detection, etc., show LocalMamba outperforms baselines like Vim and Swin Transformer under the same FLOPs.
This technology provides an efficient state space model solution for vision tasks, potentially replacing Transformer and CNN as a new mainstream architecture. It is especially beneficial for vision applications on mobile and edge devices (e.g., autonomous driving, drones) due to Mamba's linear computational complexity. Open-source code will accelerate community research and industrial deployment.
AI chip and edge computing companies are advised to evaluate LocalMamba's hardware friendliness and consider it as a core operator for vision processing units. Vision product teams can try replacing existing backbone networks with it to reduce computational costs.
Future work should focus on LocalMamba's performance on larger datasets and downstream tasks (e.g., segmentation, video understanding), as well as automation of window size and search strategy. Additionally, integration with multimodal models is a potential direction.