arXiv:2608.03918cs.CVcs.AI2026-08

让视觉语言模型自动决定看哪段视频、何时停止,提升长视频理解效率。

When and Where to Look: Adaptive Visual Evidence Scheduling for Efficient Long Video Understanding

论文配图:When and Where to Look: Adaptive Visual Evidence Scheduling for Efficient Long Video Understanding
图 1 · 摘自论文原文
  • 根据模型输出不确定性动态调整看视频的帧数和范围。
  • 在多个数据集上比现有方法准确率更高,速度提升1.85倍以上。
  • 无需训练,适合部署在资源受限的长视频分析场景。

高效理解长视频需要视觉-语言模型(VLMs)从少量选定的稀疏视觉证据中推理。现有基于相关性的方法依赖静态一次性选择,固定帧数预算和候选池;而基于智能体的调度器虽具自适应性,但需高成本多轮推理与交互搜索。本文提出EcoFrame,一种无需训练的低开销查询自适应视觉证据调度框架。EcoFrame利用VLM的推理反馈,决定何时增加帧预算、何处搜索更多候选证据。具体而言,熵门控预算调度通过输出不确定性判断是否提前终止,或逐步扩展帧预算;注意力引导候选提议将帧级注意力转化为时间先验,实现信息密集区的密集局部搜索,同时在注意力分散时保持全局覆盖。在Video-MME、LongVideoBench和MLVU上的实验表明,EcoFrame在多种VLM骨干网络下均实现了更优的准确率-效率权衡。在Qwen2.5-VL上,平均准确率达64.4,超过BOLT的63.5,且相比AKS和BOLT提速1.85倍;相较基于智能体的A.I.R.,精度相当,但推理速度最高提升13.5倍。代码将开源于https://github.com/AK-DREAM/EcoFrame。

原文摘要 · Abstract (English)

Efficient long-video understanding requires vision--language models (VLMs) to reason over a small number of frames selected as sparse visual evidence. Existing relevance-based methods rely on static one-shot selection with fixed frame budgets and candidate pools, while agent-based schedulers achieve adaptivity through costly multi-round reasoning and interactive search. We propose EcoFrame, a training-free framework for low-overhead query-adaptive visual evidence scheduling. EcoFrame leverages the VLM's inference feedback to determine when to increase the frame budget and where to search for additional candidate evidence. Specifically, entropy-gated budget scheduling uses output uncertainty to stop early when the current evidence is sufficient or progressively expand the frame budget otherwise. Meanwhile, attention-guided candidate proposal converts frame-level attention into a temporal prior, enabling dense local search in informative regions while preserving global coverage when attention is diffuse. Experiments on Video-MME, LongVideoBench, and MLVU demonstrate that EcoFrame achieves a better accuracy--efficiency trade-off across multiple VLM backbones. On Qwen2.5-VL, EcoFrame achieves an average accuracy of 64.4, surpassing BOLT at 63.5, while providing a $1.85\times$ speedup over AKS and BOLT. Compared with the agent-based A.I.R., EcoFrame maintains comparable accuracy with up to a $13.5\times$ inference speedup. Code will be available at https://github.com/AK-DREAM/EcoFrame.

视频理解视觉语言模型高效推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。