SceneBind: Binding What and Where Across Vision, Audio and Language
SceneBind is a full-modality scene representation method that unifies semantics and 3D spatial understanding across vision, audio, and language. Existing full-modality encoders excel at instance-level semantics (what is present) but lack explicit spatial structure (where it is). SceneBind represents each scene as semantic-spatial entities, combining global semantic embeddings with object-centric semantic-spatial slots to explicitly capture object-level semantics, spatial attributes, and uncertainty. SceneBind Matching is a semantic-spatial matching scheme that integrates global scene similarity with object alignment, supporting cross-modal scene retrieval and object localization. For training and evaluation, the authors curated a new real-world binaural audio-visual dataset with structured semantic and spatial annotations, and proposed a training protocol that aligns cross-modal semantic and spatial signals. SceneBind is compatible with large-scale pre-trained semantic encoders, adding only a few extra tokens for lightweight spatial modeling. It achieves state-of-the-art performance on scene and spatial retrieval tasks.
SceneBind is a full-modality scene representation method that unifies semantics and 3D spatial understanding across vision, audio, and language, enabling cross-modal scene retrieval and object localization through semantic-spatial entities and matching schemes, trained and evaluated on a real-world binaural audio-visual dataset.
SceneBind explicitly encodes object-level spatial attributes via semantic-spatial slots, addressing the lack of spatial structure in existing full-modality encoders. Its lightweight design (only a few extra tokens) facilitates easy integration into existing pre-trained models.
This work advances multimodal understanding from instance-level semantics to structured scene representation, potentially impacting applications requiring spatial awareness such as robotics, AR/VR, and autonomous driving.
SceneBind improves accuracy in cross-modal scene retrieval and object localization, empowering products requiring precise spatial understanding such as smart homes, virtual reality, and robot navigation.
Verifiable next signal: whether SceneBind is integrated into mainstream multimodal models (e.g., CLIP or ImageBind) or spawns downstream applications (e.g., scene editing or navigation).