用音频和镜头结构提升视频片段定位精度
SMART: Shot-Aware Multimodal Video Moment Retrieval with Audio-Enhanced MLLM
- 融合音视频特征，按镜头压缩令牌保留关键时间细节
- 在Charades-STA上提升1.61%的定位准确率
- 适合需要精细时间定位的多模态视频任务
视频片段定位旨在根据自然语言查询，在未剪辑视频中定位特定时间段。尽管现有方法结合传统技术与多模态大模型取得进展，但仍依赖粗粒度时间理解与单一视觉模态，限制复杂视频表现。为此，我们提出基于多模态大模型的SMART框架，融合音频线索并利用镜头级时间结构。SMART通过整合音视频特征，并采用【镜头感知令牌压缩】，有选择地保留每个镜头内的高信息量令牌，减少冗余同时保持细粒度时间细节。此外，优化提示设计以更好利用音视频线索。在Charades-STA与QVHighlights数据集上的评估显示，SMART显著优于现有方法，于Charades-STA上实现[email protected]提升1.61%、[email protected]提升2.59%。
原文摘要 · Abstract (English)
Video Moment Retrieval is a task in video understanding that aims to localize a specific temporal segment in an untrimmed video based on a natural language query. Despite recent progress in moment retrieval from videos using both traditional techniques and Multimodal Large Language Models (MLLM), most existing methods still rely on coarse temporal understanding and a single visual modality, limiting performance on complex videos. To address this, we introduce \textit{S}hot-aware \textit{M}ultimodal \textit{A}udio-enhanced \textit{R}etrieval of \textit{T}emporal \textit{S}egments (SMART), an MLLM-based framework that integrates audio cues and leverages shot-level temporal structure. SMART enriches multimodal representations by combining audio and visual features while applying \textbf{Shot-aware Token Compression}, which selectively retains high-information tokens within each shot to reduce redundancy and preserve fine-grained temporal details. We also refine prompt design to better utilize audio-visual cues. Evaluations on Charades-STA and QVHighlights show that SMART achieves significant improvements over state-of-the-art methods, including a 1.61\% increase in [email protected] and 2.59\% gain in [email protected] on Charades-STA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。