TempCloze: Can Video-LLMs Identify the Missing Middle?
TempCloze is a video cloze benchmark for evaluating visual temporal reasoning in Video-LLMs. It contains 1,521 filtered videos from seven sources, mainly long-take and egocentric videos. Models must identify the true missing middle clip from four candidates given beginning and ending clips. Distractors are constructed along three dimensions: Semantic, Alignment, and Progression. Evaluation of 10 proprietary and 21 open-source Video-LLMs reveals Alignment as the primary bottleneck.
Researchers introduced TempCloze, a benchmark designed to reduce linguistic shortcuts in temporal reasoning evaluation for Video-LLMs. The benchmark uses 1,521 videos and requires models to select the correct middle clip from four candidates. Distractors test semantic content, temporal alignment, and event progression. Testing 31 models showed that temporal alignment is the main weakness, even when models can recognize plausible content and progression.
The benchmark isolates visual temporal reasoning by using same-source distractors and shared scenes/objects to minimize appearance cues. The finding that Alignment is the primary bottleneck suggests current Video-LLMs struggle with precise temporal ordering, not just semantic understanding. Error pattern and sensitivity analyses on TempCloze-Mixed and TempCloze-Hard with four representative models are planned to further dissect failure modes.
This work highlights a gap in video understanding models that could affect applications requiring precise temporal reasoning, such as video editing, surveillance, and autonomous systems. The benchmark provides a standardized way to measure and compare temporal reasoning capabilities across proprietary and open-source models, potentially guiding future model development.
For companies developing video AI products, TempCloze offers a way to benchmark and differentiate models on temporal reasoning. Improved temporal alignment could unlock new use cases in video summarization, anomaly detection, and interactive video understanding, potentially increasing product value and market competitiveness.
Future work may focus on improving temporal alignment in Video-LLMs, possibly through better training data or architectural changes. The benchmark could become a standard evaluation tool, and follow-up studies may explore how candidate order and context length affect performance. Watch for model updates that specifically address temporal alignment.