DART通过动态路由让模型智能区分复杂与简单视频查询,提升零样本定位准确率。
DART: Difficulty-Adaptive Routing for Zero-Shot Video Temporal Grounding

- 根据查询难度自动选择快速或慢速路径,复杂任务启用分步推理机制
- 在Charades-STA和ActivityNet上比最强基线提升3.5点mIoU,仅用1/7帧数
- 适合需要高效处理复杂时序理解的视频定位场景
零样本视频时序定位(VTG)旨在无需特定训练的情况下,从未剪辑视频中定位自然语言查询所描述的事件。现有方法依赖帧-查询特征匹配,对简单事件有效,但在需理解时序顺序与因果结构的复杂多阶段查询面前表现不佳,形成所谓的“推理差距”。本文提出DART(难度自适应路由),通过将难度感知路由与大视觉语言模型中的结构化推理结合,弥合该差距。查询条件决定的行列式点过程(DPP)双重作用:筛选多样且相关的关键帧作为时序证据,并以谱熵作为难度指标。简单查询走快速路径直接预测,复杂查询则进入慢速路径,经由时序标记提示分解为全局事件分析、逐帧时序角色标注与边界提取。在Charades-STA和ActivityNet Captions数据集上,DART在相同分布与多种分布外设置下均达最优零样本性能,相比最强基线提升最高3.5点mIoU,同时使用超过7倍更少的帧数。项目主页见 https://dart-vtg.github.io/。
原文摘要 · Abstract (English)
Zero-shot video temporal grounding (VTG) localizes events in untrimmed videos from natural language queries without task-specific training. Existing methods rely on frame-query feature matching, which suffices for simple events but struggles with complex multi-stage queries that require understanding temporal ordering and causal structure -- a disparity we call the reasoning gap. We propose DART (Difficulty-Adaptive Routing for Temporal Grounding), which bridges this gap by coupling difficulty-aware routing with structured reasoning in large vision-language models. A query-conditioned Determinantal Point Process (DPP) serves a dual role: selecting diverse, query-relevant keyframes as temporal evidence, and providing spectral entropy as a difficulty indicator. Simple queries are routed to a Fast path for direct prediction, while complex queries follow a Slow path with Temporal Markup Prompting, which decomposes localization into global event analysis, per-frame temporal role annotation, and boundary extraction. On Charades-STA and ActivityNet Captions, DART achieves state-of-the-art zero-shot performance across both identically distributed and multiple out-of-distribution settings, improving mIoU by up to 3.5 points over the strongest baseline while using over 7 times fewer frames. The project homepage is available at https://dart-vtg.github.io/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。