arXiv:2510.17023cs.CVcs.MM2025-10ICCV被引 14

用多模态大模型增强查询,精准定位视频时间片段。

Enrich and Detect: Video Temporal Grounding with Multimodal LLMs

  • 先将查询扩展为含上下文的增强句,再定位时间边界。
  • 在多个基准上超越现有方法,零样本表现更优。
  • 适合需要高精度视频理解的应用场景。

我们提出ED-VTG,一种利用多模态大语言模型进行细粒度视频时间定位的方法。该方法通过两阶段流程:首先将自然语言查询转化为包含缺失信息和提示的增强句,以辅助定位;其次使用轻量级解码器基于上下文表示预测准确的时间边界。为减少噪声与幻觉影响,模型采用多实例学习目标,动态选择每个样本的最佳查询版本进行训练。实验表明,该方法在多种时间视频定位和段落定位基准上均达到当前最优性能,显著优于所有已有基于大模型的方法,在零样本评估中也保持明显优势。

原文摘要 · Abstract (English)

We introduce ED-VTG, a method for fine-grained video temporal grounding utilizing multi-modal large language models. Our approach harnesses the capabilities of multimodal LLMs to jointly process text and video, in order to effectively localize natural language queries in videos through a two-stage process. Rather than being directly grounded, language queries are initially transformed into enriched sentences that incorporate missing details and cues to aid in grounding. In the second stage, these enriched queries are grounded, using a lightweight decoder, which specializes at predicting accurate boundaries conditioned on contextualized representations of the enriched queries. To mitigate noise and reduce the impact of hallucinations, our model is trained with a multiple-instance-learning objective that dynamically selects the optimal version of the query for each training sample. We demonstrate state-of-the-art results across various benchmarks in temporal video grounding and paragraph grounding settings. Experiments reveal that our method significantly outperforms all previously proposed LLM-based temporal grounding approaches and is either superior or comparable to specialized models, while maintaining a clear advantage against them in zero-shot evaluation scenarios.

视频定位多模态大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。