用Mamba+LLM提升视频时间定位精度
MLVTG: Mamba-Based Feature Alignment and LLM-Driven Purification for Multi-Modal Video Temporal Grounding
- 用Vision Mamba替代Transformer建模时序，减少冗余注意力
- 通过冻结LLM提取语义先验，无须微调即增强对齐效果
- 在多个数据集上达到顶尖性能，适合视频理解研究者
视频时间定位（VTG）旨在定位与自然语言查询对应的视频片段，是视频理解中的基础且具有挑战性任务。现有基于Transformer的方法常面临冗余注意力和多模态对齐不佳的问题。为此，我们提出MLVTG框架，包含两个核心模块：MambaAligner与LLMRefiner。MambaAligner采用堆叠的Vision Mamba块作为主干，替代Transformer以建模时序依赖并提取稳健的视频表征，实现多模态对齐。LLMRefiner利用预训练大型语言模型（LLM）的特定冻结层，隐式传递语义先验，无需微调即可增强多模态对齐。该双对齐策略结合结构化状态空间动态建模时序与文本先验净化语义，实现更精确的定位。在QVHighlights、Charades-STA和TVSum上的大量实验表明，MLVTG达到当前最优性能，显著优于现有基线。
原文摘要 · Abstract (English)
Video Temporal Grounding (VTG), which aims to localize video clips corresponding to natural language queries, is a fundamental yet challenging task in video understanding. Existing Transformer-based methods often suffer from redundant attention and suboptimal multi-modal alignment. To address these limitations, we propose MLVTG, a novel framework that integrates two key modules: MambaAligner and LLMRefiner. MambaAligner uses stacked Vision Mamba blocks as a backbone instead of Transformers to model temporal dependencies and extract robust video representations for multi-modal alignment. LLMRefiner leverages the specific frozen layer of a pre-trained Large Language Model (LLM) to implicitly transfer semantic priors, enhancing multi-modal alignment without fine-tuning. This dual alignment strategy, temporal modeling via structured state-space dynamics and semantic purification via textual priors, enables more precise localization. Extensive experiments on QVHighlights, Charades-STA, and TVSum demonstrate that MLVTG achieves state-of-the-art performance and significantly outperforms existing baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。