用注意力机制实现测试时高效长视频帧选择，无需训练即可提升理解精度。
Efficient Frame Selection for Long Videos at Test Time with Attention-Based MLLM Selectors
- 基于MLLM的注意力特征动态生成帧相关性评分，无需自回归解码。
- 32帧预算下在Video-MME上比均匀采样高6.4分，超越已有训练型方法。
- 轻量级模型（20亿参数）通用性强，适配不同主干和任务，无需重训。
使用多模态大语言模型（MLLM）理解长视频需从数千帧中筛选出精炼帧集，但选帧又依赖对视频的理解，形成循环依赖。本文发现：在验证阶段选定的提取层中，跨模态注意力已能提供与查询相关的帧证据，无需自回归生成。据此提出DAFS（动态注意力驱动的预算感知帧选择），一种无需训练的帧选择器。仅20亿参数的轻量级MLLM选择器，通过查询条件聚合将选定层注意力转化为相关性分数，实现帧间无解码比较。针对选择器自身上下文限制，将候选帧池大小与每帧令牌预算联合优化建模为离散优化问题，由动态规划求解。在32帧预算下，该方法在Video-MME上相比均匀采样最高提升6.4分，且在匹配帧数下优于先前训练型选择器，同时在不同选择器和回答器骨干网络、跨任务间具备泛化能力，无需再训练。
原文摘要 · Abstract (English)
Understanding long videos with multimodal large language models (MLLMs) requires selecting a compact set of frames from thousands of candidates, yet identifying the right frames seemingly requires understanding the video first. We resolve this circular dependency with a simple observation: cross-modal attention at validation-selected extraction layers in MLLMs already provides query-relevant frame evidence without requiring autoregressive generation. We exploit this property to build DAFS (Dynamic Attention-based Budget-aware Frame Selection), a training-free frame selector. A lightweight MLLM selector, even with only 2B parameters, can extract frame-level evidence by converting selected-layer attention into relevance scores through query-conditioned aggregation. This enables cross-frame comparison without autoregressive decoding. To handle the selector's own context constraint, we formulate the joint allocation of candidate pool size and per-frame token budget as a discrete optimization problem solved by dynamic programming. Under a 32-frame budget, our selector improves over uniform sampling by up to 6.4 points on Video-MME and outperforms prior training-based selectors under matched frame budgets, while generalizing across selector and answerer backbones, and across tasks, without retraining.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。