让视频理解更精准:按问题需求选关键帧,不增加计算负担
QuoTA: Query-oriented Token Assignment via CoT Query Decouple for Long Video Comprehension
- 根据问题相关性预先分配帧重要性,指导视觉令牌选择
- 在6个基准上平均提升3.2%性能,保持相同令牌预算
- 无需重新训练,可直接接入现有视频语言模型
近期长视频理解方法通常通过基于注意力分布的视觉令牌剪枝来缓解视觉冗余。然而,现有方法在解码器层采用事后低响应令牌剪枝,忽略了视觉令牌与指令(查询)之间的输入级语义关联。本文提出QuoTA,一种事前、免训练的模块化方法,通过查询导向的帧级重要性评估,扩展现有大视频语言模型(LVLMs)进行视觉令牌分配。查询导向的令牌选择至关重要,它能将视觉处理对齐任务需求,优化令牌预算利用并保留语义相关内容。具体而言:(i) QuoTA基于查询相关性策略分配帧级重要性分数,实现解码器层跨模态交互前的一次性视觉令牌分配;(ii) 通过思维链(Chain-of-Thoughts)推理解耦查询,提升基于LVLM的帧重要性评分精度;(iii) QuoTA具备即插即用特性,可无缝扩展至现有LVLM。大量实验表明,在使用LLaVA-Video-7B时,引入QuoTA在六个基准(包括Video-MME和MLVU)上平均性能提升3.2%,同时维持与基线相同的视觉令牌预算。代码已开源于https://github.com/MAC-AutoML/QuoTA。
原文摘要 · Abstract (English)
Recent advances in long video understanding typically mitigate visual redundancy through visual token pruning based on attention distribution. However, while existing methods employ post-hoc low-response token pruning in decoder layers, they overlook the input-level semantic correlation between visual tokens and instructions (query). In this paper, we propose QuoTA, an ante-hoc training-free modular that extends existing large video-language models (LVLMs) for visual token assignment based on query-oriented frame-level importance assessment. The query-oriented token selection is crucial as it aligns visual processing with task-specific requirements, optimizing token budget utilization while preserving semantically relevant content. Specifically, (i) QuoTA strategically allocates frame-level importance scores based on query relevance, enabling one-time visual token assignment before cross-modal interactions in decoder layers, (ii) we decouple the query through Chain-of-Thoughts reasoning to facilitate more precise LVLM-based frame importance scoring, and (iii) QuoTA offers a plug-and-play functionality that extends to existing LVLMs. Extensive experimental results demonstrate that implementing QuoTA with LLaVA-Video-7B yields an average performance improvement of 3.2% across six benchmarks (including Video-MME and MLVU) while operating within an identical visual token budget as the baseline. Codes are open-sourced at https://github.com/MAC-AutoML/QuoTA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。