arXiv:2503.09027cs.CV2025-03被引 2

不用时间戳,用语义分割视频事件,定位更准。

Measure Twice, Cut Once: A Semantic-Oriented Approach to Video Temporal Localization with Video LLMs

  • 用特殊标记分段并分类视频内容,替代生成时间戳
  • 在多个数据集上超越传统方法,最高提升12.3%准确率
  • 适合需要精准定位事件的视频理解场景

通过自然语言查询定位视频中的事件是视频模型的重要能力。现有方法多依赖视频大模型生成事件边界时间戳,但时间戳信息量低,难以发挥大模型预训练的语义理解优势。本文提出一种无时间戳、以语义为核心的框架MeCo,通过两个生成式任务和一个判别式任务微调视频大模型。首先引入结构标记生成任务,使模型基于查询识别视频的时间结构,生成一系列特殊标记(结构标记),将视频划分为连续片段,并分类为目标事件或背景过渡。为增强事件片段的精细识别,进一步设计查询聚焦的描述生成任务,提取可被结构标记有效利用的细粒度事件语义。最后引入基于对比学习的结构标记对齐模块,将每个结构标记与对应视频片段关联,实现整体时间分割,并直接输出目标事件片段用于定位。在多种时间定位任务上的实验表明,所提框架持续优于依赖边界时间戳的方法,验证了语义驱动路径的有效性。

原文摘要 · Abstract (English)

Temporally localizing user-queried events through natural language is a crucial capability for video models. Recent methods predominantly adapt video LLMs to generate event boundary timestamps for temporal localization tasks, which struggle to leverage LLMs' pre-trained semantic understanding capabilities due to the uninformative nature of timestamp outputs. In this work, we explore a timestamp-free, semantic-oriented framework that fine-tunes video LLMs using two generative learning tasks and one discriminative learning task. We first introduce a structural token generation task that enables the video LLM to recognize the temporal structure of input videos based on the input query. Through this task, the video LLM generates a sequence of special tokens, called structural tokens, which partition the video into consecutive segments and categorize them as either target events or background transitions. To enhance precise recognition of event segments, we further propose a query-focused captioning task that enables the video LLM to extract fine-grained event semantics that can be effectively utilized by the structural tokens. Finally, we introduce a structural token grounding module driven by contrastive learning to associate each structural token with its corresponding video segment, achieving holistic temporal segmentation of the input video and readily yielding the target event segments for localization. Extensive experiments across diverse temporal localization tasks demonstrate that our proposed framework, MeCo, consistently outperforms methods relying on boundary timestamp generation, highlighting the potential of a semantic-driven approach for temporal localization with video LLMs \footnote{Code available at https://github.com/pangzss/MeCo.

视频定位语义理解视频LLM结构标记

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。