arXiv:2510.02778cs.CV2025-10被引 6

自适应选择视频关键帧,兼顾重要性与多样性,提升长视频理解准确率。

AdaRD-key: Adaptive Relevance-Diversity Keyframe Sampling for Long-form Video understanding

  • 根据查询动态调整关键帧采样,融合相关性与视觉多样性
  • 在长视频基准上优于现有方法,尤其对复杂查询效果显著
  • 无需训练、实时运行,可直接接入现有视觉语言模型

长时视频理解对视觉-语言模型仍是重大挑战,因其时间跨度长、信息密度高。当前多数多模态大模型依赖均匀采样,常忽略关键片段,导致回答错误。现有关键帧选择方法或采用固定间隔抑制冗余,或强调视觉多样性而忽视查询相关性。本文提出AdaRD-Key,一种无需训练的查询驱动关键帧采样模块,通过统一的关联性-多样性最大体积(RD-MV)目标,结合查询相关的相关性得分与对数行列式多样性项,生成既信息丰富又不重复的帧。针对查询与视频对齐弱的情况,引入轻量级相关性感知门控机制,自动切换至仅多样性的模式,增强覆盖范围且无需额外标注。该方法计算高效(单卡实时运行),兼容现有视觉语言模型,即插即用。在LongVideoBench和Video-MME上的实验表明其在长视频理解任务中达到领先性能。代码已开源。

原文摘要 · Abstract (English)

Understanding long-form videos remains a significant challenge for vision--language models (VLMs) due to their extensive temporal length and high information density. Most current multimodal large language models (MLLMs) rely on uniform sampling, which often overlooks critical moments, leading to incorrect responses to queries. In parallel, many keyframe selection approaches impose rigid temporal spacing: once a frame is chosen, an exclusion window suppresses adjacent timestamps to reduce redundancy. While effective at limiting overlap, this strategy frequently misses short, fine-grained cues near important events. Other methods instead emphasize visual diversity but neglect query relevance. We propose AdaRD-Key, a training-free keyframe sampling module for query-driven long-form video understanding. AdaRD-Key maximizes a unified Relevance--Diversity Max-Volume (RD-MV) objective, combining a query-conditioned relevance score with a log-determinant diversity component to yield informative yet non-redundant frames. To handle broad queries with weak alignment to the video, AdaRD-Key employs a lightweight relevance-aware gating mechanism; when the relevance distribution indicates weak alignment, the method seamlessly shifts into a diversity-only mode, enhancing coverage without additional supervision. Our pipeline is training-free, computationally efficient (running in real time on a single GPU), and compatible with existing VLMs in a plug-and-play manner. Extensive experiments on LongVideoBench and Video-MME demonstrate state-of-the-art performance, particularly on long-form videos. Code available at https://github.com/Xian867/AdaRD-Key.

视频理解关键帧采样多模态实时推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。