通过灵活选词提升长视频理解效率，显著降低计算成本。
FlexSelect: Flexible Token Selection for Efficient Long Video Understanding
- 基于跨模态注意力识别关键视频片段，无需额外训练
- 在多个基准上实现最高9倍加速，保持高精度
- 可插入多种视频大模型，适配长视频分析任务
长视频理解对视频大语言模型（VideoLLMs）带来巨大计算与内存挑战。本文提出FlexSelect，一种灵活高效的视频标记选择策略。该方法利用参考变换器层的跨模态注意力模式，识别并保留最具语义重要性的内容。包含两个核心组件：(1) 无需训练的标记排序流程，通过忠实的跨模态注意力权重评估每个视频标记的重要性；(2) 训练有素的轻量级选择器，学习复制这些排序并过滤冗余标记。该通用方法可无缝集成至多种VideoLLM架构，如LLaVA-Video、InternVL和Qwen-VL，作为即插即用模块扩展其时序上下文长度。实验表明，FlexSelect在VideoMME、MLVU、LongVB和LVBench等多个长视频基准上均取得显著性能提升，并实现大幅加速（例如在LLaVA-Video-7B模型上最高达9倍），展现出高效处理长视频理解的巨大潜力。
原文摘要 · Abstract (English)
Long-form video understanding poses a significant challenge for video large language models (VideoLLMs) due to prohibitively high computational and memory demands. In this paper, we propose FlexSelect, a flexible and efficient token selection strategy for processing long videos. FlexSelect identifies and retains the most semantically relevant content by leveraging cross-modal attention patterns from a reference transformer layer. It comprises two key components: (1) a training-free token ranking pipeline that leverages faithful cross-modal attention weights to estimate each video token's importance, and (2) a rank-supervised lightweight selector that is trained to replicate these rankings and filter redundant tokens. This generic approach can be seamlessly integrated into various VideoLLM architectures, such as LLaVA-Video, InternVL and Qwen-VL, serving as a plug-and-play module to extend their temporal context length. Empirically, FlexSelect delivers strong gains across multiple long-video benchmarks including VideoMME, MLVU, LongVB, and LVBench. Moreover, it achieves significant speed-ups (for example, up to 9 times on a LLaVA-Video-7B model), highlighting FlexSelect's promise for efficient long-form video understanding. Project page available at: https://yunzhuzhang0918.github.io/flex_select
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。