改进DETR模型，让视频时间定位更准
Sim-DETR: Unlock DETR for Temporal Sentence Grounding
- 通过约束查询间注意力和增加帧对齐，优化检测过程
- 在多个数据集上显著提升定位准确率，最高达14.6%提升
- 适合研究视频理解与多模态定位的学者参考
时间句子定位旨在识别视频中与给定文本查询精确对应的时间段，通常采用检测变压器（DETR）方法。然而我们发现，常规用于提升DETR性能的策略在此任务中不仅无效，甚至会降低表现。系统分析表明其根本原因在于：(1) 相似目标时刻间的查询冲突；(2) 全局语义与局部定位之间的内部查询矛盾。基于此，我们提出一个简单而强大的基线模型Sim-DETR，仅在解码器层做两项微小修改：(1) 根据语义与位置重叠度，约束查询间的自注意力；(2) 增加查询到帧的对齐机制，以弥合全局与局部上下文。实验表明，Sim-DETR成功释放了DETR在时间句子定位中的全部潜力，为未来研究提供了强有力的基准。
原文摘要 · Abstract (English)
Temporal sentence grounding aims to identify exact moments in a video that correspond to a given textual query, typically addressed with detection transformer (DETR) solutions. However, we find that typical strategies designed to enhance DETR do not improve, and may even degrade, its performance in this task. We systematically analyze and identify the root causes of this abnormal behavior: (1) conflicts between queries from similar target moments and (2) internal query conflicts due to the tension between global semantics and local localization. Building on these insights, we propose a simple yet powerful baseline, Sim-DETR, which extends the standard DETR with two minor modifications in the decoder layers: (1) constraining self-attention between queries based on their semantic and positional overlap and (2) adding query-to-frame alignment to bridge the global and local contexts. Experiments demonstrate that Sim-DETR unlocks the full potential of DETR for temporal sentence grounding, offering a strong baseline for future research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。