通过识别关键词，让模型更懂语义，精准定位视频片段
See, Rank, and Filter: Important Word-Aware Clip Filtering via Scene Understanding for Moment Retrieval and Highlight Detection
- 用MMLM分析文本关键词，动态筛选视频片段
- 在MR和HD任务上超越现有方法，显著提升准确率
- 适合需要细粒度理解的视频检索与摘要场景
视频片段检索（MR）与精彩时刻检测（HD）旨在根据自然语言查询定位相关视频片段和关键亮点。然而，现有方法将整个文本查询与视频片段视为黑箱，忽略了单个词语的重要性，制约了上下文理解。本文提出一种新方法，通过识别查询中的重要词语，实现细粒度的视频片段过滤。该方法利用多模态大语言模型（MLLM）进行图像-文本场景理解，并引入特征增强模块（FEM）提取查询中的关键信息，以及基于排名的过滤模块（RFM）迭代优化视频片段的筛选结果，使其更贴合重要词语语义。大量实验表明，本方法在MR与HD任务上均显著优于当前最优方法，性能全面领先。代码已公开：https://github.com/VisualAIKHU/SRF。
原文摘要 · Abstract (English)
Video moment retrieval (MR) and highlight detection (HD) with natural language queries aim to localize relevant moments and key highlights in a video clips. However, existing methods overlook the importance of individual words, treating the entire text query and video clips as a black-box, which hinders contextual understanding. In this paper, we propose a novel approach that enables fine-grained clip filtering by identifying and prioritizing important words in the query. Our method integrates image-text scene understanding through Multimodal Large Language Models (MLLMs) and enhances the semantic understanding of video clips. We introduce a feature enhancement module (FEM) to capture important words from the query and a ranking-based filtering module (RFM) to iteratively refine video clips based on their relevance to these important words. Extensive experiments demonstrate that our approach significantly outperforms existing state-of-the-art methods, achieving superior performance in both MR and HD tasks. Our code is available at: https://github.com/VisualAIKHU/SRF.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。