用对话大模型零样本实现视频时间定位，无需标注数据。
ChatVTG: Video Temporal Grounding via Chat with Video Dialogue Large Language Models
- 用视频对话大模型生成多粒度片段描述，匹配查询做粗定位。
- 在三个主流数据集上超越现有零样本方法，最高提升12.3%。
- 适合缺乏标注数据的视频理解场景，尤其适用于零样本任务。
视频时间定位（VTG）旨在从无剪辑视频中定位与自然语言查询对应的具体片段。现有方法主要依赖监督学习和大量标注数据，成本高且易引入人为偏见。为此，我们提出ChatVTG，一种利用视频对话大模型进行零样本视频时间定位的新方法。ChatVTG通过视频对话大模型生成多粒度片段描述，并将其与给定查询匹配以实现粗粒度时间定位，避免了配对标注数据的需求。为进一步提升定位精度，采用时段精修策略优化细粒度描述提案。在三个主流VTG数据集（Charades-STA、ActivityNet-Captions、TACoS）上的广泛实验表明，ChatVTG显著优于当前零样本方法。
原文摘要 · Abstract (English)
Video Temporal Grounding (VTG) aims to ground specific segments within an untrimmed video corresponding to the given natural language query. Existing VTG methods largely depend on supervised learning and extensive annotated data, which is labor-intensive and prone to human biases. To address these challenges, we present ChatVTG, a novel approach that utilizes Video Dialogue Large Language Models (LLMs) for zero-shot video temporal grounding. Our ChatVTG leverages Video Dialogue LLMs to generate multi-granularity segment captions and matches these captions with the given query for coarse temporal grounding, circumventing the need for paired annotation data. Furthermore, to obtain more precise temporal grounding results, we employ moment refinement for fine-grained caption proposals. Extensive experiments on three mainstream VTG datasets, including Charades-STA, ActivityNet-Captions, and TACoS, demonstrate the effectiveness of ChatVTG. Our ChatVTG surpasses the performance of current zero-shot methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。