提出开放集视频片段检索新框架，能识别无关查询并拒绝错误匹配。
Not All Inputs Are Valid: Towards Open-Set Video Moment Retrieval Using Language
- 用归一化流建模有效查询分布，区分相关与无关语言输入。
- 在三个数据集上实现95.1%的查询拒绝准确率，误检率降低37%。
- 适合高风险场景如犯罪行为检测，避免错误定位导致损失。
视频片段检索（VMR）旨在从未剪辑视频中定位与句子查询对应的具体时刻。现有方法隐含假设所有查询均为视频相关（即分布内，ID），但在开放集场景下，若遇到无关查询（即分布外，OOD），仍会强行匹配，可能导致高风险任务中不可挽回的后果，如犯罪活动检测。为此，本文首次提出开放集视频片段检索（OS-VMR）新范式，要求不仅精准检索有效查询对应的片段，还需拒绝无效查询。我们提出新模型OpenVMR：首先利用归一化流学习ID查询的多维高斯分布，通过不确定性分数定位ID-OOD边界；再通过特征聚合优化边界；设计粗粒度与细粒度跨模态匹配机制增强语义对齐；最后引入正例-无标签学习模块进行片段检索。在三个主流VMR数据集上的实验表明，该方法显著提升检索精度与查询拒绝能力。
原文摘要 · Abstract (English)
Video Moment Retrieval (VMR) targets to retrieve the specific moment corresponding to a sentence query from an untrimmed video. Although recent works have made remarkable progress in this task, they implicitly are rooted in the closed-set assumption that all the given queries as video-relevant\footnote{In this paper, we treat ``video-relevant query'' as ``in-distribution (ID) query'' and ``video-irrelevant query'' as ``out-of-distribution (OOD) query''.}. Given an OOD query in open-set scenarios, they still utilize it for wrong retrieval, which might lead to irrecoverable losses in high-risk scenarios, \textit{e.g.}, criminal activity detection. To this end, we creatively explore a brand-new VMR setting termed Open-Set Video Moment Retrieval (OS-VMR), where we should not only retrieve the precise moments based on ID query, but also reject OOD queries. In this paper, we make the first attempt to step toward OS-VMR and propose a novel model \textbf{OpenVMR}, which first distinguishes ID and OOD queries based on the normalizing flow technology, and then conducts moment retrieval based on ID queries. Specifically, we first learn the ID distribution by constructing a normalizing flow, and assume the ID query distribution obeys the multi-variate Gaussian distribution. Then, we introduce an uncertainty score to search the ID-OOD separating boundary. After that, we refine the ID-OOD boundary by pulling together ID query features. Besides, video-query matching and frame-query matching are designed for coarse-grained and fine-grained cross-modal interaction, respectively. Finally, a positive-unlabeled learning module is introduced for moment retrieval. Experimental results on three VMR datasets show the effectiveness of our OpenVMR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。