arXiv:2412.09601cs.CVcs.AI2024-12被引 16

通过多次精修时间边界,提升视频时序定位精度。

TimeRefine: Temporal Grounding with Time Refining Video LLM

  • 将定位任务转为逐步精修的迭代过程,先粗略预测再修正偏移。
  • 在ActivityNet和Charades-STA上分别提升3.6%和5.0%的mIoU。
  • 适合作为插件集成到主流视频大模型中,提升时序理解能力。

视频时序定位旨在根据文本提示定位视频中的相关时间边界。近期工作尝试通过视频大模型(Video LLM)进行下一时间戳预测来实现该任务,但仅依赖时间标记预测仍难以准确对齐。本文提出TimeRefine方法,从两个方面解决此问题:首先,将时序定位重构为时序精修任务——模型先给出粗略预测,再通过预测偏移量逐步修正至目标段;该过程可重复多轮,使模型自我迭代优化定位精度。其次,引入辅助预测头,对偏离真实边界的预测施加更大惩罚,鼓励模型生成更接近真实的时间区间。该方法为即插即用设计,可集成于多数基于大模型的时序定位框架。实验表明,TimeRefine在ActivityNet和Charades-STA数据集上分别取得3.6%和5.0%的mIoU提升。代码与预训练模型将公开发布。

原文摘要 · Abstract (English)

Video temporal grounding aims to localize relevant temporal boundaries in a video given a textual prompt. Recent work has focused on enabling Video LLMs to perform video temporal grounding via next-token prediction of temporal timestamps. However, accurately localizing timestamps in videos remains challenging for Video LLMs when relying solely on temporal token prediction. Our proposed TimeRefine addresses this challenge in two ways. First, instead of directly predicting the start and end timestamps, we reformulate the temporal grounding task as a temporal refining task: the model first makes rough predictions and then refines them by predicting offsets to the target segment. This refining process is repeated multiple times, through which the model progressively self-improves its temporal localization accuracy. Second, to enhance the model's temporal perception capabilities, we incorporate an auxiliary prediction head that penalizes the model more if a predicted segment deviates further from the ground truth, thus encouraging the model to make closer and more accurate predictions. Our plug-and-play method can be integrated into most LLM-based temporal grounding approaches. The experimental results demonstrate that TimeRefine achieves 3.6% and 5.0% mIoU improvements on the ActivityNet and Charades-STA datasets, respectively. Code and pretrained models will be released.

时序定位视频大模型精修机制自迭代

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。