arXiv:2605.12954cs.CVcs.AI2026-05

AdaFocus通过动态采样与按需检索,高效理解长视频内容。

AdaFocus: Adaptive Relevance-Diversity Sampling with Zero-Cache Look-back for Efficient Long Video Understanding

论文配图:AdaFocus: Adaptive Relevance-Diversity Sampling with Zero-Cache Look-back for Efficient Long Video Understanding
图 1 · 摘自论文原文
  • 自适应采样生成紧凑视频预览,查询不确定时自动切换全局聚类
  • 仅在模型不自信时才从磁盘按需加载高分辨率帧,降低33倍视觉标记消耗
  • 无需内存缓存,适合资源受限环境下的长视频推理任务

长视频理解受限于僵化的单次编码范式:现有方法要么以高内存和延迟代价密集编码视频,要么过度压缩为稀疏帧集,导致下游推理所需的细粒度证据不可逆丢失。为此,我们提出AdaFocus,将长视频理解重新构想为渐进式证据获取过程。其核心包含两个紧密耦合组件:首先,查询感知的自适应相关性-多样性采样器(AdaRD)生成紧凑且信息丰富的视频预览,当查询缺乏可靠局部定位时自动切换至全局聚类;其次,摒弃全量帧缓存,引入不确定性触发的精化机制,仅在模型信心不足时按需从磁盘检索高分辨率证据,采用零缓存输入输出设计。这使被丢弃的视觉细节变为可按需恢复的证据,而无需承担全量预加载成本。在七个标准长视频基准上的实验表明,相比传统密集编码,AdaFocus在保持更高性能的同时,显著提升效率-精度权衡:例如在VideoMME上提升2.59分准确率,在Charades-STA上提升8.39分mIoU,同时视觉标记消耗减少约33倍,并完全消除内存帧预缓存需求。结果表明,渐进式预览结合零缓存证据精化是可扩展多媒体推理的有效范式。

原文摘要 · Abstract (English)

Long video understanding is heavily bottlenecked by a rigid one-shot paradigm: existing methods either densely encode videos at prohibitive memory and latency costs, or aggressively compress them into sparse frame sets that irreversibly discard fine-grained evidence needed for downstream reasoning. Consequently, current models struggle to simultaneously balance temporal coverage, visual details, and computational efficiency. We propose AdaFocus, an efficient framework that rethinks long-video understanding as progressive evidence acquisition rather than one-pass encoding. AdaFocus relies on two tightly coupled components. First, a Query-Aware Adaptive Relevance-Diversity sampler (AdaRD) produces a compact yet informative video preview, adaptively switching to global clustering when the query lacks reliable local grounding. Second, instead of caching exhaustive frame sequences in memory, AdaFocus introduces an uncertainty-triggered refinement mechanism. It performs targeted look-back only when the model is not confident, retrieving high-resolution evidence directly from disk via a zero-cache I/O design. This turns discarded visual details from an irreversible loss into on-demand recoverable evidence without paying the cost of exhaustive preloading. Experiments on seven standard long-video benchmarks show that AdaFocus delivers a substantially better efficiency-accuracy trade-off than strong baselines. Compared with conventional dense encoding, AdaFocus achieves improved task performance (e.g., +2.59 accuracy on VideoMME, +8.39 mIoU on Charades-STA over single-pass inference) while reducing visual token consumption by ~33x and eliminating the need for in-memory frame pre-caching through its zero-cache disk retrieval design. These findings suggest that progressive preview combined with zero-cache evidence refinement is a highly effective paradigm for scalable multimedia reasoning.

长视频理解自适应采样零缓存高效推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。