EcoFrame: Adaptive Visual Evidence Scheduling for Efficient Long Video Understanding
EcoFrame is a training-free framework for query-adaptive visual evidence scheduling in vision-language models (VLMs) for long video understanding. It uses entropy-gated budget scheduling to stop early or expand frame budget based on output uncertainty, and attention-guided candidate proposal to convert frame-level attention into a temporal prior for dense local search. Experiments on Video-MME, LongVideoBench, and MLVU show improved accuracy-efficiency trade-off across multiple VLM backbones, including Qwen2.5-VL.
Researchers introduced EcoFrame, a training-free framework that improves the efficiency of long video understanding by adaptively scheduling visual evidence. Unlike static one-shot selection or costly multi-round agent-based methods, EcoFrame uses the VLM's own inference feedback: entropy-gated budget scheduling dynamically adjusts the number of frames processed based on output uncertainty, while attention-guided candidate proposal uses frame-level attention to guide where to search for additional evidence. Evaluated on Video-MME, LongVideoBench, and MLVU, EcoFrame achieves a better accuracy-efficiency trade-off across multiple VLM backbones, with results reported for Qwen2.5-VL.
EcoFrame introduces a novel combination of entropy-based early stopping and attention-based temporal prior for adaptive frame selection. The entropy gate allows the model to decide when enough evidence has been gathered, reducing unnecessary computation, while the attention-guided proposal focuses search on informative temporal regions without losing global coverage. This training-free approach can be applied to existing VLMs without retraining, offering a plug-and-play efficiency gain.
Efficient long video understanding is critical for applications like video surveillance, content moderation, and autonomous driving. EcoFrame's ability to reduce computational cost while maintaining accuracy could accelerate the deployment of VLMs in resource-constrained environments or real-time systems. Its training-free nature lowers the barrier for adoption by existing VLM-based products.
By reducing the number of frames processed without sacrificing accuracy, EcoFrame can lower inference costs and latency for long video understanding services. This makes it attractive for cloud-based video analysis platforms, edge devices, and any application where per-query cost is a concern. It could also enable new use cases that were previously too expensive to run at scale.
Next signals to watch include: (1) integration of EcoFrame into commercial VLM APIs or open-source libraries; (2) extensions to other modalities (e.g., audio-visual); (3) benchmarks on real-world long video tasks beyond academic datasets; (4) comparisons with other emerging adaptive inference methods; and (5) potential patent filings or startup formation around adaptive visual scheduling.