通过逻辑关系动态搜索关键帧,提升长视频理解准确性
Logic-in-Frames: Dynamic Keyframe Search via Visual Semantic-Logical Verification for Long Video Understanding
- 基于视觉语义逻辑关系迭代优化关键帧采样
- 在多个基准上达到当前最优性能,显著提升问答准确率
- 适合需要精准时序推理的长视频分析任务
长视频理解常依赖密集采样帧描述或端到端特征选择,但这些方法往往忽略文本查询与视觉元素间的逻辑关系。受计算限制,需粗粒度帧子采样,类似‘大海捞针’。为此,我们提出一种语义驱动的搜索框架,将关键帧选择重构为视觉语义逻辑搜索。系统定义四种基本逻辑依赖:空间共现、时间邻近、属性依赖和因果顺序。这些关系通过迭代精炼过程动态更新帧采样分布,实现对特定查询需求的上下文感知关键帧识别。该方法在人工标注基准上取得新的最先进性能。应用于下游视频问答任务时,在LongVideoBench和Video-MME上均实现最佳性能提升,验证了其在弥合文本查询与视觉-时序推理间逻辑鸿沟方面的有效性。代码将公开。
原文摘要 · Abstract (English)
Understanding long video content is a complex endeavor that often relies on densely sampled frame captions or end-to-end feature selectors, yet these techniques commonly overlook the logical relationships between textual queries and visual elements. In practice, computational constraints necessitate coarse frame subsampling, a challenge analogous to "finding a needle in a haystack." To address this issue, we introduce a semantics-driven search framework that reformulates keyframe selection under the paradigm of Visual Semantic-Logical Search. Specifically, we systematically define four fundamental logical dependencies: 1) spatial co-occurrence, 2) temporal proximity, 3) attribute dependency, and 4) causal order. These relations dynamically update frame sampling distributions through an iterative refinement process, enabling context-aware identification of semantically critical frames tailored to specific query requirements. Our method establishes new SOTA performance on the manually annotated benchmark in key-frame selection metrics. Furthermore, when applied to downstream video question-answering tasks, the proposed approach demonstrates the best performance gains over existing methods on LongVideoBench and Video-MME, validating its effectiveness in bridging the logical gap between textual queries and visual-temporal reasoning. The code will be publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。