通过内外双重线索定位长视频关键片段，提升问答准确率。
VideoDetective: Clue Hunting via both Extrinsic Query and Intrinsic Relevance for Long Video Understanding
- 构建视觉-时间关联图，融合查询与视频内在结构
- 在VideoMME-long上最高提升7.5%准确率
- 适合需要精确定位长视频内容的研究者
长视频理解对多模态大语言模型（MLLMs）仍具挑战，受限于上下文窗口，需精准定位稀疏的查询相关片段。现有方法多仅依赖查询本身进行线索定位，忽视视频内在结构及片段间差异化的相关性。为此，我们提出VideoDetective框架，结合查询-片段相关性与片段间亲和性，实现长视频问答中的有效线索挖掘。具体地，将视频划分为多个片段，并基于视觉相似性和时间邻近性构建视觉-时间亲和图。通过假设-验证-优化循环，估计已观察片段对查询的相关性并传播至未见片段，生成全局相关性分布，指导最关键的片段定位，以稀疏观测完成最终回答。实验表明，该方法在主流MLLM上均取得显著提升，在VideoMME-long上最高达7.5%准确率增长。代码已开源。
原文摘要 · Abstract (English)
Long video understanding remains challenging for multimodal large language models (MLLMs) due to limited context windows, which necessitate identifying sparse query-relevant video segments. However, existing methods predominantly localize clues based solely on the query, overlooking the video's intrinsic structure and varying relevance across segments. To address this, we propose VideoDetective, a framework that integrates query-to-segment relevance and inter-segment affinity for effective clue hunting in long-video question answering. Specifically, we divide a video into various segments and represent them as a visual-temporal affinity graph built from visual similarity and temporal proximity. We then perform a Hypothesis-Verification-Refinement loop to estimate relevance scores of observed segments to the query and propagate them to unseen segments, yielding a global relevance distribution that guides the localization of the most critical segments for final answering with sparse observation. Experiments show our method consistently achieves substantial gains across a wide range of mainstream MLLMs on representative benchmarks, with accuracy improvements of up to 7.5% on VideoMME-long. Our code is available at https://videodetective.github.io/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。