解决视频片段检索中从描述语句到搜索查询的性能下降问题
Beyond Caption-Based Queries for Video Moment Retrieval
- 通过修改公开数据集的文本查询构建新基准，发现语言不完整与多片段查询两大挑战
- 提出解码器-查询坍缩现象是模型泛化差的关键原因，改进架构提升活跃查询数
- 在搜索查询上提升14.82% mAP_m，多片段查询上提升21.83%，适合视频理解研究者
本文研究现有视频片段检索（VMR）方法，特别是DETR架构，在使用描述性查询训练却在搜索查询上评估时的性能退化问题。为此，我们对三个公开数据集（HD-EPIC、YouCook2、ActivityNet-Captions）的文本查询进行了修改，构建了三个新基准。分析揭示两大泛化挑战：(i) 语言间隙，源于搜索查询的语言信息不足；(ii) 多片段间隙，来自单片段到多片段查询的转变。同时识别出架构中的关键问题——解码器-查询坍缩，是导致对多片段实例泛化能力差的主要原因。通过架构改进有效增加活跃解码器查询数量，显著缓解该问题。大量实验表明，该方法在搜索查询上提升14.82% mAP_m，多片段搜索查询上提升21.83%。代码、模型与数据已发布于项目主页：https://davidpujol.github.io/beyond-vmr/
原文摘要 · Abstract (English)
In this work, we investigate the degradation of existing VMR methods, particularly of DETR architectures, when trained on caption-based queries but evaluated on search queries. For this, we introduce three benchmarks by modifying the textual queries in three public VMR datasets -- i.e., HD-EPIC, YouCook2 and ActivityNet-Captions. Our analysis reveals two key generalization challenges: (i) A language gap, arising from the linguistic under-specification of search queries, and (ii) a multi-moment gap, caused by the shift from single-moment to multi-moment queries. We also identify a critical issue in these architectures -- an active decoder-query collapse -- as a primary cause of the poor generalization to multi-moment instances. We mitigate this issue with architectural modifications that effectively increase the number of active decoder queries. Extensive experiments demonstrate that our approach improves performance on search queries by up to 14.82% mAP_m, and up to 21.83% mAP_m on multi-moment search queries. The code, models and data are available in the project webpage: https://davidpujol.github.io/beyond-vmr/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。