SAM 2: Segment Anything in Images and Videos - A Unified Foundation Model for Image and Video Segmentation with Significantly Improved Real-Time Processing and Interaction Efficiency
In August 2024, Meta released SAM 2, unifying image and video segmentation into a streaming memory Transformer architecture. Through a data engine, the largest video segmentation dataset was collected. In video segmentation, it achieves higher accuracy with only one-third the interactions of previous methods; in image segmentation, it is 6x faster and more accurate than SAM. The model, dataset, and code are open-sourced.
SAM 2 unifies image and video segmentation into a single foundation model, employing a streaming memory Transformer for real-time video processing. Its data engine iteratively improves the model and data through user interactions, collecting the largest video segmentation dataset to date. In video segmentation, SAM 2 achieves higher accuracy with fewer interactions; in image segmentation, it is 6x faster and more accurate than SAM. This unified paradigm will significantly reduce the development and deployment costs of visual segmentation systems, driving applications in autonomous driving, video editing, medical imaging, and more.
The core innovation of SAM 2 is unifying image and video segmentation into a promptable streaming memory Transformer architecture. The model includes an image encoder, a memory attention module, and a mask decoder. The memory attention module stores and retrieves embeddings from historical frames to achieve temporal consistency, supporting real-time video processing. Training employs a large-scale data engine that collects annotations through a user interaction loop, resulting in a dataset with over 50 million masks. Evaluations show that on video benchmarks like DAVIS and YouTube-VOS, SAM 2 achieves higher mAP with 3x fewer interactions; on image benchmarks like COCO, it achieves 2.4 higher AP and is 6x faster than SAM. Limitations include room for improvement in handling fast motion and small objects.
SAM 2 unifies image and video segmentation, greatly simplifying the development process of visual AI systems. In autonomous driving, it can simultaneously handle single-frame detection and multi-frame tracking; in video editing, it enables real-time object segmentation and replacement; in medical imaging, it supports frame-by-frame segmentation of 3D scans. Its open-source strategy will accelerate industry adoption and lower the technical barrier for small and medium enterprises.
It is recommended that computer vision teams immediately evaluate SAM 2 to replace existing segmentation pipelines, especially in scenarios like video editing and autonomous driving perception. Vertical industry solutions can be developed based on the open-source model, such as real-time defect segmentation in industrial quality inspection. Investment attention should be paid to third-party toolchains that may emerge from Meta's open ecosystem.
Focus on the deployment efficiency of SAM 2 on edge devices (e.g., phones, drones), as well as fine-tuned versions for specific domains (e.g., medical, remote sensing). Its data engine model may be adopted by other vision tasks. Observe the model's stability in long videos and complex occlusion scenarios, and the impact of commercial licensing terms on the industry.