用事件锚点选帧，提升长视频理解效率与精度
Event-Anchored Frame Selection for Effective Long-Video Understanding
- 按视觉相似性分段，每段选最相关帧作锚点
- 在三个基准上分别提升4.7%、4.9%、8.8%准确率
- 无需训练，可直接接入现有多模态模型
海量帧冗余与有限上下文窗口使长视频理解中高效帧选择至关重要。现有方法采用扁平采样，将视频视为无结构帧集合。本文提出事件锚点帧选择（EFS），一种分层、事件感知的管道。利用自监督DINO嵌入，先将视频流划分为视觉同质的时间片段，作为语义事件的代理。在每个事件内，选取最符合查询的帧作为锚点。这些锚点作为结构先验，引导全局优化阶段，采用自适应最大边际相关性（MMR）策略。该流程确保最终关键帧集联合优化事件覆盖、查询相关性和视觉多样性。作为免训练、即插即用模块，EFS可无缝集成至现成大视觉-语言模型（LVLMs），在挑战性视频理解基准上取得显著提升：应用于LLaVA-Video-7B时，在VideoMME、LongVideoBench和MLVU上分别提升4.7%、4.9%和8.8%准确率。
原文摘要 · Abstract (English)
Massive frame redundancy and limited context window make efficient frame selection crucial for long-video understanding with large vision-language models (LVLMs). Prevailing approaches, however, adopt a flat sampling paradigm which treats the video as an unstructured collection of frames. In this paper, we introduce Event-Anchored Frame Selection (EFS), a hierarchical, event-aware pipeline. Leveraging self-supervised DINO embeddings, EFS first partitions the video stream into visually homogeneous temporal segments, which serve as proxies for semantic events. Within each event, it then selects the most query-relevant frame as an anchor. These anchors act as structural priors that guide a global refinement stage using an adaptive Maximal Marginal Relevance (MMR) scheme. This pipeline ensures the final keyframe set jointly optimizes for event coverage, query relevance, and visual diversity. As a training-free, plug-and-play module, EFS can be seamlessly integrated into off-the-shelf LVLMs, yielding substantial gains on challenging video understanding benchmarks. Specifically, when applied to LLaVA-Video-7B, EFS improves accuracy by 4.7%, 4.9%, and 8.8% on VideoMME, LongVideoBench, and MLVU, respectively.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。