Seeing Before Synthesizing: VLM-Guided Transition Event Discovery for Weakly-Supervised Dense Video Captioning
A paper titled 'Seeing Before Synthesizing: VLM-Guided Transition Event Discovery for Weakly-Supervised Dense Video Captioning' was published on arXiv on 2026-09-03. The paper proposes a framework called SBS that uses a vision-language model to generate frame-level narratives for inter-event gaps and detect transitions from semantic variation. It refines inter-event temporal masks by blending temporal midpoint with semantic change point and selecting width maximizing vision-language alignment. Experiments on ActivityNet Captions and YouCook2 demonstrate state-of-the-art performance in captioning and localization.
The paper introduces Seeing Before Synthesizing (SBS), a framework for weakly-supervised dense video captioning. It addresses limitations of prior work that synthesizes auxiliary transition captions via LLM without visual grounding and rigidly assigns them to every inter-event gap. SBS uses a VLM to generate frame-level narratives for inter-event gaps, detects transitions from semantic variation, and refines temporal masks by blending temporal midpoint with semantic change point and selecting width maximizing vision-language alignment. Experiments on ActivityNet Captions and YouCook2 show state-of-the-art performance in both captioning and localization.
The method leverages a vision-language model to produce frame-level narratives, enabling detection of transition events based on semantic variation rather than fixed temporal assumptions. It refines temporal masks by combining temporal midpoint with semantic change point and optimizing width for vision-language alignment, improving localization accuracy.
This research advances weakly-supervised dense video captioning, which could reduce annotation costs for video understanding tasks. State-of-the-art results on ActivityNet Captions and YouCook2 suggest potential for applications in video summarization, content indexing, and automated captioning.
Improved weakly-supervised dense video captioning could lower data labeling costs and enable more scalable video content analysis for media, education, and surveillance industries.
Future work may explore applying SBS to other video understanding benchmarks or integrating it with larger vision-language models. Observing whether the approach generalizes to real-world untrimmed videos and whether it influences subsequent weakly-supervised video captioning research will be key signals.