arXiv:2512.06673cs.CV2025-12被引 1

用检测器提速视频时空定位,效率提升三倍以上

Detector-Empowered Video Large Language Model for Efficient Spatio-Temporal Grounding

  • 将空间定位交给预训练检测器,单次完成而非逐帧推理
  • 在HC-STVG数据集上达43.1% m_vIoU,速度14.33 FPS
  • 适合需要高效视频理解的实时应用或部署场景

多模态大语言模型正从通用视频理解向细粒度的时空定位(STVG)任务演进。现有方法分为直接定位和候选选择两类:前者随查询时间跨度增长导致解码成本线性上升,后者依赖昂贵的候选生成。为此,我们提出DEViL——一种基于检测器增强的视频-语言模型。其核心思想是将密集空间定位任务卸载给可并行的预训练检测器。具体地,将查询语义编码为检测器兼容的参考标记,实现单次通过的空间定位;再设计时间一致性正则化,保持目标跨帧一致性。该方法避免了长坐标解码和复杂候选管道。大量实验表明,DEViL在HC-STVG上达到43.1% m_vIoU,同时实现14.33 FPS的高效率,且保留了原有大模型的通用推理能力。

原文摘要 · Abstract (English)

Multimodal large language models (MLLMs) are rapidly expanding from general video understanding to finer-grained understanding such as spatio-temporal video grounding (STVG) and reasoning. In these tasks, an MLLM must localize the user-queried target in time and space and take the results as evidence for reasoning. Existing MLLM methods mainly follow two paradigms: (1) Direct Localization, which outputs STVG results with extra alignment modules or specialized decoders; and (2) Candidate-based Selection, which first constructs tube-level candidates and then selects the relevant one by an MLLM. However, both suffer from a serious efficiency bottleneck: the former incurs linearly growing decoding cost as the queried temporal span increases, while the latter relies on costly candidate construction. To break this bottleneck, we propose DEViL, a detector-empowered Video-LLM with a simple key idea: offloading dense spatial grounding from the MLLM to a fully parallelizable, well-trained detector. Specifically, DEViL distills the query into a detector-compatible reference-semantic token, which replaces the detector's text embedding to enable spatial grounding in a single pass. Then, we design temporal consistency regularization to match objects across frames and enforce their coherence over time. In this way, DEViL avoids long coordinate decoding and heavy candidate pipelines. Extensive experiments show that DEViL achieves strong performance (43.1% m_vIoU on HC-STVG) with superior efficiency (14.33 FPS), while preserving the general reasoning capacity of the MLLM backbone.

视频理解时空定位效率优化检测器融合

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。