用文本条件重建视频目标管,提升弱监督时空定位精度
TubeRMC: Tube-conditioned Reconstruction with Mutual Constraints for Weakly-supervised Spatio-Temporal Video Grounding
- 以视频目标管为条件,通过三重约束重构语言线索
- 在两个公开数据集上超越现有方法,显著减少误检和跟踪断裂
- 适合做弱监督视频理解与多模态定位的研究者参考
时空视频定位(STVG)旨在未剪辑视频中定位与给定语言查询对应的时空目标管。该任务挑战大,需复杂的视觉-语言理解与时空推理。近期工作探索弱监督设置以避免依赖细粒度标注(如边界框或时间戳)。然而,现有方法通常采用简单的后期融合方式,生成独立于文本的目标管,常导致目标识别失败和跟踪不一致。为此,我们提出管状条件重建与互约束框架(TubeRMC),先用预训练视觉定位模型生成文本条件候选管,再通过管状条件重建及时空约束进行精修。设计了从时序、空间、时空三个角度的重建策略,每种策略配备管状条件重建器,利用时空管作为条件重构查询中的关键线索。进一步引入空间与时序提议间的互约束机制,提升重建质量。TubeRMC在两个公开基准VidSTG和HCSTVG上表现更优。可视化显示其有效缓解了目标识别错误与跟踪不一致问题。
原文摘要 · Abstract (English)
Spatio-Temporal Video Grounding (STVG) aims to localize a spatio-temporal tube that corresponds to a given language query in an untrimmed video. This is a challenging task since it involves complex vision-language understanding and spatiotemporal reasoning. Recent works have explored weakly-supervised setting in STVG to eliminate reliance on fine-grained annotations like bounding boxes or temporal stamps. However, they typically follow a simple late-fusion manner, which generates tubes independent of the text description, often resulting in failed target identification and inconsistent target tracking. To address this limitation, we propose a Tube-conditioned Reconstruction with Mutual Constraints (\textbf{TubeRMC}) framework that generates text-conditioned candidate tubes with pre-trained visual grounding models and further refine them via tube-conditioned reconstruction with spatio-temporal constraints. Specifically, we design three reconstruction strategies from temporal, spatial, and spatio-temporal perspectives to comprehensively capture rich tube-text correspondences. Each strategy is equipped with a Tube-conditioned Reconstructor, utilizing spatio-temporal tubes as condition to reconstruct the key clues in the query. We further introduce mutual constraints between spatial and temporal proposals to enhance their quality for reconstruction. TubeRMC outperforms existing methods on two public benchmarks VidSTG and HCSTVG. Further visualization shows that TubeRMC effectively mitigates both target identification errors and inconsistent tracking.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。