arXiv:2604.17422cs.CVcs.MM2026-04被引 1

根据查询动态选择视频关键帧,提升长视频理解效率与准确性。

Where to Focus: Query-Modulated Multimodal Keyframe Selection for Long Video Understanding

论文配图:Where to Focus: Query-Modulated Multimodal Keyframe Selection for Long Video Understanding
图 1 · 摘自论文原文
  • 用三个轻量专家流分别处理视觉细节、场景语义和字幕叙事。
  • 基于查询意图动态分配权重,抑制无关模态噪声。
  • 无需训练,兼容主流多模态大模型,适合长视频分析任务。

长视频理解对多模态大语言模型(MLLMs)仍是重大挑战,因其需处理密集帧序列导致计算成本过高。现有方法通常依赖单一视觉指标(如CLIP相似度)或静态融合启发式得分,这种‘一刀切’策略常失效:仅靠视觉的指标对剧情驱动查询无效,而盲目融合文本得分则在纯视觉任务中引入严重‘模态噪声’。为此,我们提出Q-Gate,一个即插即用、无需训练的框架,将关键帧选择视为动态模态路由问题。将检索过程解耦为三个轻量级专家流:视觉定位(局部细节)、全局匹配(场景语义)、上下文对齐(字幕驱动叙事)。关键在于,Q-Gate引入查询调制门控机制,利用LLM的上下文推理能力评估查询意图,动态分配各专家注意力权重。该机制智能激活必要模态,同时‘屏蔽’无关模态,从而最大化信噪比。在LongVideoBench和Video-MME上对多个MLLM骨干网络的大量实验表明,Q-Gate显著优于当前最优基线。通过有效抑制模态特异性噪声,提供了一种鲁棒且高度可解释的可扩展视频推理方案。

原文摘要 · Abstract (English)

Long video understanding remains a formidable challenge for Multimodal Large Language Models (MLLMs) due to the prohibitive computational cost of processing dense frame sequences. Prevailing solutions, which select a keyframe subset, typically rely on either a single visual-centric metric (e.g., CLIP similarity) or a static fusion of heuristic scores. This ``one-size-fits-all'' paradigm frequently fails: visual-only metrics are ineffective for plot-driven narrative queries, while indiscriminately incorporating textual scores introduces severe ``modal noise'' for purely visual tasks. To break this bottleneck, we propose Q-Gate, a plug-and-play and training-free framework that treats keyframe selection as a dynamic modality routing problem. We decouple the retrieval process into three lightweight expert streams: Visual Grounding for local details, Global Matching for scene semantics, and Contextual Alignment for subtitle-driven narratives. Crucially, Q-Gate introduces a Query-Modulated Gating Mechanism that leverages the in-context reasoning of an LLM to assess the query's intent and dynamically allocate attention weights across the experts. This mechanism intelligently activates necessary modalities while ``muting'' irrelevant ones, thereby maximizing the signal-to-noise ratio. Extensive experiments on LongVideoBench and Video-MME across multiple MLLM backbones demonstrate that Q-Gate substantially outperforms state-of-the-art baselines. By effectively suppressing modality-specific noise, it provides a robust, highly interpretable solution for scalable video reasoning.

视频理解关键帧选择多模态提示工程

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。