用三步法从长视频中精准选帧,提升问答准确率
Ground, Cover, and Refine: Evidence-Centric Frame Selection for Long-Video Question Answering

- 分三步:定位关键事件、补充视觉锚点、替换弱帧
- 在32帧预算下,比最强基线高1.93~2.54个百分点
- 无需训练,适配多种模型,适合长视频问答场景
长视频问答需在数千帧中有限视觉令牌预算下提取稀疏但关键的证据。现有方法或单次选取查询相关帧,或仅依赖时间标记文本作为检索引导,导致两个问题:一是选帧集中在局部相关峰值,预算耗尽后无法补救遗漏证据;二是文本与视觉证据对齐弱。我们提出GCR,一种免训练框架,将固定预算帧选择视为联合证据整理问题。Ground将时间标记文本转化为时间事件,选出与查询相关的实际帧锚点,并将每段事件文本映射到其时间对齐的帧上。Cover通过直接视觉锚点补充已定位事件,并采用全局最大边际相关性保留多样上下文。Refine重新审视被忽略的时间区域,仅当真实帧中位数提供更高证据价值时,才替换最弱可替换帧。GCR保持固定数量且按时间排序的帧,无需视觉语言模型训练或架构修改。在LongVideoBench和Video-MME上,使用三个7B骨干模型及8、32、64帧预算的实验表明,该方法在长视频问答中持续提升性能。以7B LLaVA-OV为骨干、32帧预算为例,分别取得64.25%和62.15%准确率,超越最强复现基线2.54和1.93个百分点。
原文摘要 · Abstract (English)
Long-video question answering requires identifying sparse yet critical evidence from videos containing thousands of frames under a constrained visual-token budget. Existing methods either select query-aware frames in a single pass or rely on timestamped text solely as retrieval guidance, leading to two key limitations. First, selected frames tend to cluster around local relevance peaks, and once the budget is exhausted, omitted evidence cannot be recovered. Second, textual and visual evidence remain weakly aligned. We propose GCR, a training-free framework that casts fixed-budget frame selection as a joint evidence curation problem. Ground converts timestamped text into temporal events, selects query-relevant real frame anchors, and renders each event text onto its temporally aligned frame. Cover supplements grounded events with direct visual anchors for complementary visual evidence and applies global maximal marginal relevance to preserve diverse context. Refine revisits omitted temporal regions and replaces the weakest revisable context frame with a real-frame medoid---but only when the medoid offers greater evidence value. GCR maintains a fixed number of chronologically ordered frames and requires no VLM training or architectural modification. Experiments on LongVideoBench and Video-MME, across three 7B backbones and frame budgets of 8, 32, and 64, demonstrate consistent improvements in long-video QA. With the 7B LLaVA-OV backbone and 32 frames, GCR achieves 64.25% and 62.15% on the two benchmarks, outperforming the strongest reproduced baselines by 2.54 and 1.93 percentage points, respectively.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。