用轻量适配器让2D模型学会视频时空定位,小数据下表现优异。
Unlocking the Potential of Grounding DINO in Videos: Parameter-Efficient Adaptation for Limited-Data Spatial-Temporal Localization

- 冻结预训练模型,仅添加约1000万参数的轻量适配器注入时序感知能力。
- 在小规模数据集上达到领先性能,且在跨域数据集上保持良好泛化性。
- 适合标注成本高、数据稀缺的视频理解任务,如医疗或工业场景。
时空视频定位(STVG)旨在动态视频片段中定位目标对象。现有全量训练方法对数据需求极高,但大规模STVG数据难以获取:帧级边界框与复杂时间语言对齐的标注成本极高,尤其在专业视频领域。因此,传统模型在数据有限时严重过拟合,而零样本基础模型又缺乏任务特定的时间感知能力。为此,我们提出ST-GD框架,将预训练2D视觉-语言模型(如Grounding DINO)高效适配至视频任务。为避免破坏预训练先验,该方法冻结主干模型,仅引入约1000万可训练参数的轻量适配器,并设计新型时序解码器实现边界预测,从而自然缓解数据稀缺问题。实验表明,ST-GD在有限数据的HC-STVG v1/v2基准上表现卓越,同时在VidSTG数据集上保持强泛化能力,验证其在严苛小数据约束下的视频理解潜力。
原文摘要 · Abstract (English)
Spatio-temporal video grounding (STVG) aims to localize queried objects within dynamic video segments. Prevailing fully-trained approaches are notoriously data-hungry. However, gathering large-scale STVG data is exceptionally challenging: dense frame-level bounding boxes and complex temporal language alignments are prohibitively expensive to annotate, especially for specialized video domains. Consequently, conventional models suffer from severe overfitting on these inherently limited datasets, while zero-shot foundational models lack the task-specific temporal awareness needed for precise localization. To resolve this small-data challenge, we introduce ST-GD, a data-efficient framework that adapts pre-trained 2D visual-language models (e.g., Grounding DINO) to video tasks. To avoid destroying pre-trained priors on small datasets, ST-GD keeps the base model frozen and strategically injects lightweight adapters (~10M trainable parameters) to instill spatio-temporal awareness, alongside a novel temporal decoder for boundary prediction. This design naturally counters data scarcity. Consequently, ST-GD excels in data-scarce scenarios, achieving highly competitive performance on the limited-scale HC-STVG v1/v2 benchmarks, while maintaining robust generalization on the VidSTG dataset. This validates ST-GD as a powerful paradigm for complex video understanding under strict small-data constraints.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。