用Mamba提升图像模型看视频的能力，兼顾精度与效率
MoMa: Modulating Mamba for Adapting Image Foundation Models to Video Recognition
- 引入序贯调制操作，统一建模时空特征
- 在多个视频数据集上超越现有方法，计算量更低
- 适合想高效迁移图像模型到视频任务的研究者
视频理解需有效建模时空动态。尽管图像基础模型（IFMs）在图像理解中表现优异，但现有参数高效微调方法多将空间与时间信息分开处理，难以捕捉视频的完整动态。本文提出MoMa，通过将Mamba的可选择状态空间建模整合进IFMs，实现全时空建模。设计新型SeqMod操作，将时空信息注入预训练模型，不破坏原有特征。结合分治调制架构，显著提升视频理解性能，同时保持计算高效。大量实验表明，该方法在多个视频基准上表现更优，且计算成本更低。
原文摘要 · Abstract (English)
Video understanding is a complex challenge that requires effective modeling of spatial-temporal dynamics. With the success of image foundation models (IFMs) in image understanding, recent approaches have explored parameter-efficient fine-tuning (PEFT) to adapt IFMs for video. However, most of these methods tend to process spatial and temporal information separately, which may fail to capture the full intricacy of video dynamics. In this paper, we propose MoMa, an efficient adapter framework that achieves full spatial-temporal modeling by integrating Mamba's selective state space modeling into IFMs. We propose a novel SeqMod operation to inject spatial-temporal information into pre-trained IFMs, without disrupting their original features. By incorporating SeqMod into a Divide-and-Modulate architecture, MoMa enhances video understanding while maintaining computational efficiency. Extensive experiments on multiple video benchmarks demonstrate the effectiveness of MoMa, achieving superior performance with reduced computational cost.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。