让大模型精准定位视频中的关键片段，提升长视频理解能力。
LLaVA-MR: Large Language-and-Vision Assistant for Video Moment Retrieval
- 用密集帧与时间编码提取时空特征，增强视觉感知
- 通过选帧机制捕捉短时视觉和运动模式，提升定位精度
- 动态压缩令牌解决大模型上下文限制，适合视频分析研究者
多模态大语言模型（MLLM）广泛用于视觉感知、理解与推理，但长视频处理与精确时刻检索仍面临挑战，主要受限于大模型的上下文容量和粗粒度帧提取。我们提出大语言-视觉助手用于时刻检索（LLaVA-MR），利用密集帧与时间编码（DFTE）进行时空特征提取，通过有信息量的帧选择（IFS）捕捉短暂的视觉与运动模式，并采用动态令牌压缩（DTC）缓解大模型上下文限制。在Charades-STA和QVHighlights等基准上的评估显示，LLaVA-MR超越11种先进方法，在QVHighlights数据集上实现[email protected]提升1.82%、[email protected]提升1.29%。代码将在论文接收后开源。
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs) are widely used for visual perception, understanding, and reasoning. However, long video processing and precise moment retrieval remain challenging due to LLMs' limited context size and coarse frame extraction. We propose the Large Language-and-Vision Assistant for Moment Retrieval (LLaVA-MR), which enables accurate moment retrieval and contextual grounding in videos using MLLMs. LLaVA-MR combines Dense Frame and Time Encoding (DFTE) for spatial-temporal feature extraction, Informative Frame Selection (IFS) for capturing brief visual and motion patterns, and Dynamic Token Compression (DTC) to manage LLM context limitations. Evaluations on benchmarks like Charades-STA and QVHighlights demonstrate that LLaVA-MR outperforms 11 state-of-the-art methods, achieving an improvement of 1.82% in [email protected] and 1.29% in [email protected] on the QVHighlights dataset. Our implementation will be open-sourced upon acceptance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。