提出解耦时空对齐的新框架,提升视频定位精度与泛化能力
Bridging Time and Space: Decoupled Spatio-Temporal Alignment for Video Grounding

- 分离时序与空间定位任务,避免多模态模型输出混淆
- 在VidSTG数据集上平均m_vIoU从26.4提升至34.3
- 适用于需要细粒度视频理解的多任务场景
时空视频定位需基于自然语言查询,在时间与空间维度共同定位目标物体,这对多模态大模型(MLLM)构成根本挑战。本文识别出两大核心问题:一是时空对齐的纠缠性,即在同一位移输出空间中耦合异构任务;二是双域视觉标记冗余,目标对象在时间和空间上均呈现稀疏性,导致绝大多数视觉标记与查询无关。为此,提出端到端框架Bridge-STG,通过解耦时序与空间定位并保持语义连贯性。为缓解解耦带来的语义断层,设计两个关键机制:显式时序对齐(ETA)的时空语义桥接(STSB)将时序推理上下文转化为增强型桥接查询,作为稳健语义接口;查询引导的空间定位(QGSL)模块利用这些查询驱动专用空间解码器,结合多层交互式查询与正负帧采样,联合消除双域冗余。大量实验表明,Bridge-STG在多个基准上达到当前最优性能,于VidSTG上平均m_vIoU从26.4提升至34.3,并在统一多任务训练下展现出强跨任务迁移能力。
原文摘要 · Abstract (English)
Spatio-Temporal Video Grounding requires jointly localizing target objects across both temporal and spatial dimensions based on natural language queries, posing fundamental challenges for existing Multimodal Large Language Models (MLLMs). We identify two core challenges: \textit{entangled spatio-temporal alignment}, arising from coupling two heterogeneous sub-tasks within the same autoregressive output space, and \textit{dual-domain visual token redundancy}, where target objects exhibit simultaneous temporal and spatial sparsity, rendering the overwhelming majority of visual tokens irrelevant to the grounding query. To address these, we propose \textbf{Bridge-STG}, an end-to-end framework that decouples temporal and spatial localization while maintaining semantic coherence. While decoupling is the natural solution to this entanglement, it risks creating a semantic gap between the temporal MLLM and the spatial decoder. Bridge-STG resolves this through two pivotal designs: the \textbf{Spatio-Temporal Semantic Bridging (STSB)} mechanism with Explicit Temporal Alignment (ETA) distills the MLLM's temporal reasoning context into enriched bridging queries as a robust semantic interface; and the \textbf{Query-Guided Spatial Localization (QGSL)} module leverages these queries to drive a purpose-built spatial decoder with multi-layer interactive queries and positive/negative frame sampling, jointly eliminating dual-domain visual token redundancy. Extensive experiments across multiple benchmarks demonstrate that Bridge-STG achieves state-of-the-art performance among MLLM-based methods. Bridge-STG improves average m\_vIoU from $26.4$ to $34.3$ on VidSTG and demonstrates strong cross-task transfer across various fine-grained video understanding tasks under a unified multi-task training regime.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。