用因果事件建模提升视频时序定位,让大模型更懂视频结构。
TRACE: Temporal Grounding Video LLM via Causal Event Modeling
- 将视频输出建模为带时间戳、显著性分数和文本描述的事件序列。
- 在多个数据集上优于现有视频大模型,实现零样本预测。
- 适合需要精准时序定位的视频编辑与检索任务。
视频时序定位(VTG)是视频理解模型的关键能力,对视频浏览与编辑等下游任务至关重要。当前基于视频大模型的方法主要依赖自然语言生成,缺乏对视频内在结构的建模,限制了其在VTG任务中的表现。为此,本文首次提出因果事件建模框架,将视频大模型输出表示为事件序列,利用前序事件、视频输入与文本指令预测当前事件。每个事件包含时间戳、显著性分数与文本描述三部分。我们进一步设计了一种新型任务交错式视频大模型TRACE,分别处理视觉帧、时间戳、显著性分数与文本,通过不同编码器与解码头实现,并按因果事件建模框架排列任务令牌。在多个VTG任务与数据集上的大量实验表明,TRACE显著优于现有先进视频大模型。代码与模型已开源:https://github.com/gyxxyg/TRACE。
原文摘要 · Abstract (English)
Video Temporal Grounding (VTG) is a crucial capability for video understanding models and plays a vital role in downstream tasks such as video browsing and editing. To effectively handle various tasks simultaneously and enable zero-shot prediction, there is a growing trend in employing video LLMs for VTG tasks. However, current video LLM-based methods rely exclusively on natural language generation, lacking the ability to model the clear structure inherent in videos, which restricts their effectiveness in tackling VTG tasks. To address this issue, this paper first formally introduces causal event modeling framework, which represents video LLM outputs as sequences of events, and predict the current event using previous events, video inputs, and textural instructions. Each event consists of three components: timestamps, salient scores, and textual captions. We then propose a novel task-interleaved video LLM called TRACE to effectively implement the causal event modeling framework in practice. The TRACE process visual frames, timestamps, salient scores, and text as distinct tasks, employing various encoders and decoding heads for each. Task tokens are arranged in an interleaved sequence according to the causal event modeling framework's formulation. Extensive experiments on various VTG tasks and datasets demonstrate the superior performance of TRACE compared to state-of-the-art video LLMs. Our model and code are available at https://github.com/gyxxyg/TRACE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。