用短视频序列训练长视频模型，提升时间定位能力
Seq2Time: Sequential Knowledge Transfer for Video LLM Temporal Grounding
- 将图像和短视频序列转为时间结构数据，实现自监督训练
- 在YouCook2上F1提升27.6%，Charades-STA召回率增14.7%
- 适合做视频时序理解与多模态大模型研究者参考
时间感知对视频大语言模型理解长视频中的事件至关重要，支持密集视频字幕生成与统一系统中的时间定位。然而，缺乏带有详细字幕和精确时间标注的长视频数据限制了其时间感知能力。本文提出Seq2Time，一种数据驱动的训练范式，利用图像序列和短视频片段增强长视频的时间感知。通过将序列位置转化为时间标注，我们将大规模图像与片段字幕数据集转换为模拟长视频时间结构的序列，实现大量时间敏感数据的自监督训练。为实现序列到时间的知识迁移，我们引入一种新颖的时间表示，统一图像序列、片段序列与长视频中的位置信息。实验表明，该方法在YouCook2基准上F1分数提升27.6%，CIDEr得分提升44.8%；在Charades-STA基准上召回率提高14.7%，显著优于基线。
原文摘要 · Abstract (English)
Temporal awareness is essential for video large language models (LLMs) to understand and reason about events within long videos, enabling applications like dense video captioning and temporal video grounding in a unified system. However, the scarcity of long videos with detailed captions and precise temporal annotations limits their temporal awareness. In this paper, we propose Seq2Time, a data-oriented training paradigm that leverages sequences of images and short video clips to enhance temporal awareness in long videos. By converting sequence positions into temporal annotations, we transform large-scale image and clip captioning datasets into sequences that mimic the temporal structure of long videos, enabling self-supervised training with abundant time-sensitive data. To enable sequence-to-time knowledge transfer, we introduce a novel time representation that unifies positional information across image sequences, clip sequences, and long videos. Experiments demonstrate the effectiveness of our method, achieving a 27.6% improvement in F1 score and 44.8% in CIDEr on the YouCook2 benchmark and a 14.7% increase in recall on the Charades-STA benchmark compared to the baseline.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。