统一框架实现长视频精准时间定位,支持多任务端到端训练。
TimeLoc: A Unified End-to-End Framework for Precise Timestamp Localization in Long Videos
- 单阶段模型支持文本查询与多动作输出,端到端联合训练。
- 可处理超3万帧长视频,多数据集上性能领先1.1%-2.94% mAP。
- 适合需要跨任务通用定位的视频理解研究者使用。
未剪辑视频中的时间定位旨在识别特定时间戳,对视频理解至关重要但极具挑战。该任务包含动作定位、视频文本对齐、片段检索和事件边界检测等多个子任务。现有方法多针对特定任务设计,泛化能力差。本文提出TimeLoc,一个统一的端到端时间定位框架,可处理多种任务。首先,采用简单有效的单阶段定位模型,支持文本查询输入与多动作输出;其次,通过端到端联合训练视频编码器与定位模型,并引入时间分块技术,有效处理超过3万帧的长视频;第三,通过多阶段微调预训练文本编码器,显著提升文本条件下的定位性能。TimeLoc在多个基准测试中达到最新水平:在THUMOS14和EPIC-Kitchens-100上分别优于此前最佳方法1.3%和1.9% mAP,Kinetics-GEBD上提升1.1%,QVHighlights上提升2.94% mAP,且在时间视频对齐任务中表现突出(TACoS上提升11.5%,Charades-STA上提升6.7%,[email protected]指标下)。代码与模型将开源。
原文摘要 · Abstract (English)
Temporal localization in untrimmed videos, which aims to identify specific timestamps, is crucial for video understanding but remains challenging. This task encompasses several subtasks, including temporal action localization, temporal video grounding, moment retrieval, and generic event boundary detection. Existing methods in each subfield are typically designed for specific tasks and lack generalizability across domains. In this paper, we propose TimeLoc, a unified end-to-end framework for timestamp localization that can handle multiple tasks. First, our approach employs a simple yet effective one-stage localization model that supports text queries as input and multiple actions as output. Second, we jointly train the video encoder and localization model in an end-to-end manner. To efficiently process long videos, we introduce temporal chunking, enabling the handling of videos with over 30k frames. Third, we find that fine-tuning pre-trained text encoders with a multi-stage training strategy further enhances text-conditioned localization. TimeLoc achieves state-of-the-art results across multiple benchmarks: +1.3% and +1.9% mAP over previous best methods on THUMOS14 and EPIC-Kitchens-100, +1.1% on Kinetics-GEBD, +2.94% mAP on QVHighlights, and significant improvements in temporal video grounding (+11.5% on TACoS and +6.7% on Charades-STA under [email protected]). Our code and checkpoints will be released at https://github.com/sming256/TimeLoc.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。