用新型架构让大模型高效处理两小时以上视频。
Look Every Frame All at Once: Video-Ma$^2$mba for Efficient Long-form Video Understanding with Multi-Axis Gradient Checkpointing
- 用状态空间模型替代注意力机制，实现线性内存增长。
- 在单张显卡上处理超两小时视频，内存显著低于传统方法。
- 适合需要长视频理解的场景，如智能监控、视频检索。
随着视频数据规模和复杂度的增长，现有基于Transformer的大模型在处理长视频序列时面临内存与计算量的二次增长挑战。为此，我们提出Video-Ma$^2$mba，将状态空间模型（SSMs）融入Mamba-2框架，取代注意力机制，使模型在时间和内存上的需求呈线性增长，从而可高效处理长时间视频内容。此外，我们引入多轴梯度检查点（MA-GC）策略，通过在多个计算轴上仅保留关键激活值，进一步提升内存效率。实证分析表明，Video-Ma$^2$mba可在单个GPU上处理长达数百万个标记或超过两小时连续视频（1 FPS），同时精准捕捉时间动态，在长视频理解任务中显著优于现有框架。
原文摘要 · Abstract (English)
With the growing scale and complexity of video data, efficiently processing long video sequences poses significant challenges due to the quadratic increase in memory and computational demands associated with existing transformer-based Large Multi-modal Models (LMMs). To address these issues, we introduce Video-Ma$^2$mba, a novel architecture that incorporates State Space Models (SSMs) within the Mamba-2 framework, replacing the attention mechanisms. This allows the LMMs to scale linearly in terms of time and memory requirements, making it feasible to handle long-duration video content. Furthermore, we enhance the memory efficiency introducing the Multi-Axis Gradient Checkpointing (MA-GC) method, which strategically manages memory by retaining only essential activations across multiple computational axes. Our approach significantly reduces the memory footprint compared to standard gradient checkpointing. Empirical analyses show that Video-Ma$^2$mba can process extensive video sequences-equivalent to millions of tokens or over two hours of continuous sequences at 1 FPS-on a single GPU. By maintaining a detailed capture of temporal dynamics, our model improves the accuracy and relevance of responses in long video understanding tasks, demonstrating substantial advantages over existing frameworks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。