给视频帧编号，让大模型像翻漫画一样精准定位时间点。
Number it: Temporal Grounding Videos like Flipping Manga
- 为每帧视频添加数字编号，将时序定位转化为翻漫画式阅读。
- 在moment retrieval和highlight detection上分别提升6.9%和8.5%。
- 无需额外计算成本，适合想提升视频理解精度的研究者。
视频大模型在问答对话中已展现强大视觉理解能力，但在需要精确时间定位的任务——视频时序定位（VTG）上表现不佳。为此，我们提出新方法Number-Prompt（NumPro），通过为每帧视频添加唯一编号，将视频视为带编号的图像序列，使大模型能像翻漫画一样按序读取事件时间线，实现视觉内容与时间信息的精准关联。实验表明，NumPro显著提升顶级视频大模型的时序定位性能，且无需额外计算开销。在增强数据集上微调后，该方法在moment retrieval任务上达到6.9%的mIoU提升，在highlight detection任务上实现8.5%的mAP提升，创下新纪录。代码将开源。
原文摘要 · Abstract (English)
Video Large Language Models (Vid-LLMs) have made remarkable advancements in comprehending video content for QA dialogue. However, they struggle to extend this visual understanding to tasks requiring precise temporal localization, known as Video Temporal Grounding (VTG). To address this gap, we introduce Number-Prompt (NumPro), a novel method that empowers Vid-LLMs to bridge visual comprehension with temporal grounding by adding unique numerical identifiers to each video frame. Treating a video as a sequence of numbered frame images, NumPro transforms VTG into an intuitive process: flipping through manga panels in sequence. This allows Vid-LLMs to "read" event timelines, accurately linking visual content with corresponding temporal information. Our experiments demonstrate that NumPro significantly boosts VTG performance of top-tier Vid-LLMs without additional computational cost. Furthermore, fine-tuning on a NumPro-enhanced dataset defines a new state-of-the-art for VTG, surpassing previous top-performing methods by up to 6.9\% in mIoU for moment retrieval and 8.5\% in mAP for highlight detection. The code will be available at https://github.com/yongliang-wu/NumPro.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。