用现成大模型零样本定位视频片段，自动纠正查询偏差提升准确率。
Zero-shot Video Moment Retrieval via Off-the-shelf Multimodal Large Language Models
- 用LLaMA-3重写查询，消除语言偏见影响。
- 结合MiniGPT-v2生成候选片段，再用VideoChatGPT筛选最优结果。
- 无需微调，在多个数据集上超越现有零样本方法。
视频片段检索（VMR）的目标是预测与给定语言查询语义匹配的视频时间区间。现有基于多模态大模型（MLLM）的方法过度依赖昂贵的高质量数据集和耗时的微调。尽管近期研究引入了零样本设置以避免微调，但忽略了查询中的固有语言偏见，导致定位错误。为此，本文提出Moment-GPT，一种利用冻结MLLM的无调优零样本VMR流水线。首先，使用LLaMA-3对查询进行修正与重述，以缓解语言偏见；随后，设计结合MiniGPT-v2的跨度生成器，自适应生成候选时间段；最后，借助VideoChatGPT和跨度评分器，利用MLLM的视频理解能力筛选最合适的区间。所提方法在多个公开数据集（包括QVHighlights、ActivityNet-Captions和Charades-STA）上显著优于当前最先进的基于MLLM及零样本模型。
原文摘要 · Abstract (English)
The target of video moment retrieval (VMR) is predicting temporal spans within a video that semantically match a given linguistic query. Existing VMR methods based on multimodal large language models (MLLMs) overly rely on expensive high-quality datasets and time-consuming fine-tuning. Although some recent studies introduce a zero-shot setting to avoid fine-tuning, they overlook inherent language bias in the query, leading to erroneous localization. To tackle the aforementioned challenges, this paper proposes Moment-GPT, a tuning-free pipeline for zero-shot VMR utilizing frozen MLLMs. Specifically, we first employ LLaMA-3 to correct and rephrase the query to mitigate language bias. Subsequently, we design a span generator combined with MiniGPT-v2 to produce candidate spans adaptively. Finally, to leverage the video comprehension capabilities of MLLMs, we apply VideoChatGPT and span scorer to select the most appropriate spans. Our proposed method substantially outperforms the state-ofthe-art MLLM-based and zero-shot models on several public datasets, including QVHighlights, ActivityNet-Captions, and Charades-STA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。