arXiv:2607.28463cs.CV2026-07

根据问题类型智能选帧,提升长视频理解效果

VisualRouter: Query-Grounded Visual Sampling for Long Video Understanding

论文配图:VisualRouter: Query-Grounded Visual Sampling for Long Video Understanding
图 1 · 摘自论文原文
  • 按查询类型自动选择采样策略:全局用相关性+覆盖混合,局部用事件感知分段选帧
  • 在三个数据集上相比均匀采样提升5.2%~11.6%,显著优于现有无训练采样方法
  • 无需训练可直接接入,适合需要高效处理长视频的视觉语言模型应用

大型视觉语言模型(LVLMs)在视频理解方面取得显著进展,但长视频因视觉标记数量庞大且上下文窗口有限而难以处理。视觉采样通过选取信息量高的帧子集提供实用解决方案。然而,现有方法或依赖相关性感知采样导致帧冗余、时间覆盖不足,或对所有查询采用固定策略。本文提出VisualRouter,一种无需训练、可即插即用的查询引导视觉采样框架。该框架首先将查询分类为全局或局部,再应用相应采样策略。针对全局查询,采用相关性-覆盖混合策略,在保留时间覆盖的同时保持与查询相关的视觉证据;针对局部查询,采用事件感知帧选择策略,通过事件分割、段级帧分配和事件内帧选择,以有限帧数联合平衡相关性、覆盖性和多样性。实验表明,VisualRouter在多个LVLM上均优于均匀采样,在Video-MME、LongVideoBench和MLVU上分别取得5.2%、7.7%和11.6%的性能提升(使用Qwen2.5-VL-7B),且超越现有无训练采样方法。

原文摘要 · Abstract (English)

Large vision-language models (LVLMs) have achieved significant progress in video understanding, yet understanding long videos remains challenging due to the large number of visual tokens and limited context windows. Visual sampling provides a practical solution by selecting an informative subset of frames. However, existing methods typically either rely on relevance-aware sampling, leading to redundant frame selection and insufficient temporal coverage, or adopt a fixed sampling strategy regardless of query type. In this paper, we propose VisualRouter, a training-free and plug-and-play framework for query-grounded visual sampling. VisualRouter first classifies each query as either global or local and then applies the corresponding sampling strategy. For global queries, it employs a relevance-coverage hybrid strategy that preserves temporal coverage while retaining query-relevant visual evidence. For local queries, it adopts an event-aware frame selection strategy that performs event partitioning, segment-level frame allocation, and intra-event frame selection, jointly balancing relevance, coverage, and diversity with a limited number of input frames. Experiments show that VisualRouter consistently improves multiple LVLMs over uniform sampling, achieving gains of 5.2%, 7.7%, and 11.6% on Video-MME, LongVideoBench, and MLVU with Qwen2.5-VL-7B, and outperforming existing training-free visual sampling methods under the same setting.

长视频理解视觉采样多模态查询引导

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。