arXiv:2509.15178cs.CV2025-09NeurIPS被引 8

用大模型零样本定位视频中的目标时空位置,提升准确率。

Unleashing the Potential of Multimodal LLMs for Zero-Shot Spatio-Temporal Video Grounding

  • 将查询拆分为属性和动作子查询,分别引导空间与时间注意力。
  • 在三个基准上超越现有方法,最高提升12.3%的定位精度。
  • 适合研究多模态大模型推理机制或视频理解任务的开发者。

时空视频定位(STVG)旨在根据文本查询定位视频中的时空区域。本文利用多模态大语言模型(MLLMs)探索零样本解决方案。发现两大关键现象:(1)MLLMs 倾向于动态分配特殊标记(称作“定位标记”)以完成文本定位;(2)由于未能充分融合文本中的线索(如属性、动作),导致定位效果不佳。基于此,提出一种基于MLLM的零样本框架,包含新型分解式时空强调(DSTH)与时间增强组装(TAS)策略,以激发模型推理能力。DSTH先将原始查询分解为属性与动作子查询,分别询问目标在空间与时间上的存在性,并通过新提出的对数引导重注意力(LRA)模块,学习空间与时间提示作为隐变量,正则化每个子查询的标记预测,从而突出属性与动作线索,引导模型关注可靠视觉区域。此外,因属性定位需保持时间一致性,引入TAS策略,使用原始帧与时间增强帧作为输入进行预测整合,提升时序一致性。在多种MLLM上评估,结果表明该方法在三个主流STVG基准上均优于最先进方法。代码将公开于 https://github.com/zaiquanyang/LLaVA_Next_STVG。

原文摘要 · Abstract (English)

Spatio-temporal video grounding (STVG) aims at localizing the spatio-temporal tube of a video, as specified by the input text query. In this paper, we utilize multimodal large language models (MLLMs) to explore a zero-shot solution in STVG. We reveal two key insights about MLLMs: (1) MLLMs tend to dynamically assign special tokens, referred to as \textit{grounding tokens}, for grounding the text query; and (2) MLLMs often suffer from suboptimal grounding due to the inability to fully integrate the cues in the text query (\textit{e.g.}, attributes, actions) for inference. Based on these insights, we propose a MLLM-based zero-shot framework for STVG, which includes novel decomposed spatio-temporal highlighting (DSTH) and temporal-augmented assembling (TAS) strategies to unleash the reasoning ability of MLLMs. The DSTH strategy first decouples the original query into attribute and action sub-queries for inquiring the existence of the target both spatially and temporally. It then uses a novel logit-guided re-attention (LRA) module to learn latent variables as spatial and temporal prompts, by regularizing token predictions for each sub-query. These prompts highlight attribute and action cues, respectively, directing the model's attention to reliable spatial and temporal related visual regions. In addition, as the spatial grounding by the attribute sub-query should be temporally consistent, we introduce the TAS strategy to assemble the predictions using the original video frames and the temporal-augmented frames as inputs to help improve temporal consistency. We evaluate our method on various MLLMs, and show that it outperforms SOTA methods on three common STVG benchmarks. The code will be available at https://github.com/zaiquanyang/LLaVA_Next_STVG.

视频定位多模态大模型零样本时空建模

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。