用模型内部注意力证据动态选帧,让长视频理解更高效准确。
Evidence-Driven Dynamic Visual Selector for Efficient Long Video Understanding

- 基于目标模型内部注意力构建动态选帧机制,精准定位关键片段。
- 减少约50%视觉标记量,实现3.9倍端到端加速,性能更优。
- 适合需要高效处理长视频的多模态大模型应用者。
基于多模态大模型(MLLM)的长视频理解虽降低了推理成本并突破上下文长度限制,但现有方法多依赖外部代理评分器和固定规则,难以与目标模型的内在证据对齐,且无法适应视频中时空信息密度的非均匀性。本文提出细粒度动态视觉选择框架EviSelect,基于目标MLLM内部注意力证据设计。通过稀疏预填充高效探测视觉证据,构建结构化先验以指导分布感知的动态采样。具体地,利用高度压缩的视觉输入与稀疏注意力近似目标模型的注意力图,保持与完整版本的高度对齐。基于由此生成的三个互补注意力分量,设计轻量级选择器,可精确定位查询相关时间戳,并自适应调整局部采样率与空间分辨率。为实现证据驱动的时空采样,将选择器建模为随机策略,并通过GRPO在联合准确率-效率奖励下优化。通过组内相对比较奖励低视觉开销下的正确预测,促使策略根据视频信息密度动态分配计算资源。在三个长视频理解基准上,EviSelect相比现有方法表现更优,视觉标记减少约50%,端到端速度提升3.9倍。
原文摘要 · Abstract (English)
Recent advancements in MLLM-based long-form video understanding have mitigated inference-time computational cost and limited context lengths by selecting query-relevant frames. However, existing approaches predominantly rely on external proxy scorers and rigid heuristic rules, inevitably suffering from misalignment with the target MLLM's intrinsic evidence and failing to accommodate the non-uniform spatiotemporal information density. In this paper, we propose a fine-grained dynamic visual selection framework named EviSelect, grounded in the target MLLM internal attention evidence. Our method efficiently probes visual evidence via sparse prefilling as a structured prior to guide distribution-aware dynamic sampling. Specifically, we efficiently approximate attention maps of the target MLLM using highly compressed visual inputs and sparse attention, well-aligned to the full counterpart. Conditioned on three complementary attention components derived from this prior, we design a lightweight selector that not only precisely locates query-relevant timestamps but also adaptively adjusts the local sampling rate and spatial resolution. To enable evidence-conditioned spatiotemporal sampling, we formulate the selector as a stochastic policy and optimize it via GRPO under a joint accuracy--efficiency reward. By rewarding correct predictions under lower visual cost through group-relative comparisons, our method encourages the policy to allocate computation dynamically according to the information density of each video. Across three long video understanding benchmarks, EviSelect achieves superior performance compared to existing methods while reducing selected visual tokens by about 50\% and achieving a 3.9x end-to-end speedup.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。