用大模型实现跨视频类型的时间定位,支持任意时长和复杂提问。
Universal Video Temporal Grounding with Generative Multi-modal Large Language Models
- 用多模态大模型结合时间标记,精准定位视频中的时间片段。
- 自适应帧采样让模型在短片和长片上都表现稳定,准确率超现有方法。
- 适合需要精确定位视频内容的场景,如长视频问答与智能检索。
本文提出UniTime,一种通用视频时间定位计算模型,能够根据自然语言查询(如问题或描述)精确识别视频中的时间片段。不同于以往局限于特定视频领域或长度的方法,UniTime利用生成式多模态大模型(MLLM)强大的视觉-语言理解能力,有效处理不同视角、类型和长度的视频,并理解复杂语言指令。核心贡献包括:(i) 将强大多模态大模型用于视频时间定位,通过将时间戳标记与视频标记交替输入,实现精确时间输出;(ii) 通过自适应帧缩放训练,使模型可应对不同输入粒度,在短视频与长视频上均表现鲁棒;(iii) 在五个公开基准上,无论是零样本还是微调设置,均优于当前最优方法;(iv) 作为长视频问答(VideoQA)的预检索器,显著提升问答准确率,凸显其在复杂视频理解任务中的价值。
原文摘要 · Abstract (English)
This paper presents a computational model for universal video temporal grounding, which accurately localizes temporal moments in videos based on natural language queries (e.g., questions or descriptions). Unlike existing methods that are often limited to specific video domains or durations, we propose UniTime, a robust and universal video grounding model leveraging the strong vision-language understanding capabilities of generative Multi-modal Large Language Models (MLLMs). Our model effectively handles videos of diverse views, genres, and lengths while comprehending complex language queries. The key contributions include: (i) We consider steering strong MLLMs for temporal grounding in videos. To enable precise timestamp outputs, we incorporate temporal information by interleaving timestamp tokens with video tokens. (ii) By training the model to handle videos with different input granularities through adaptive frame scaling, our approach achieves robust temporal grounding for both short and long videos. (iii) Comprehensive experiments show that UniTime outperforms state-of-the-art approaches in both zero-shot and dataset-specific finetuned settings across five public temporal grounding benchmarks. (iv) When employed as a preliminary moment retriever for long-form video question-answering (VideoQA), UniTime significantly improves VideoQA accuracy, highlighting its value for complex video understanding tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。