用混合架构让模型高效理解长达一小时的视频。
Vamba: Understanding Hour-Long Videos with Hybrid Mamba-Transformers
- 引入Mamba-2块，线性复杂度编码视频帧，无需压缩
- 单卡可处理超1024帧视频，内存减少50%以上，速度翻倍
- 在长视频任务上提升4.3%准确率，适合长视频理解场景
当前基于Transformer的大规模多模态模型难以处理时长超过一小时的视频输入，因其因果自注意力机制具有二次复杂度，导致训练与推理成本高昂。现有基于标记压缩的方法虽减少视频标记数，但常造成信息丢失，且对极长序列仍不高效。本文提出一种正交方向的混合Mamba-Transformer模型VAMBA，采用Mamba-2块以线性复杂度编码视频标记，无需任何标记压缩。VAMBA可在单张GPU上处理超过1024帧（640×360）的视频，而传统Transformer模型仅能处理256帧。在长视频输入下，VAMBA在训练和推理中至少降低50%的GPU内存占用，每步训练速度接近翻倍。实验表明，相比先前高效的视频大模型，VAMBA在挑战性的长视频理解基准LVBench上准确率提升4.3%，并在多种长短视频理解任务中保持优异性能。
原文摘要 · Abstract (English)
State-of-the-art transformer-based large multimodal models (LMMs) struggle to handle hour-long video inputs due to the quadratic complexity of the causal self-attention operations, leading to high computational costs during training and inference. Existing token compression-based methods reduce the number of video tokens but often incur information loss and remain inefficient for extremely long sequences. In this paper, we explore an orthogonal direction to build a hybrid Mamba-Transformer model (VAMBA) that employs Mamba-2 blocks to encode video tokens with linear complexity. Without any token reduction, VAMBA can encode more than 1024 frames (640$\times$360) on a single GPU, while transformer-based models can only encode 256 frames. On long video input, VAMBA achieves at least 50% reduction in GPU memory usage during training and inference, and nearly doubles the speed per training step compared to transformer-based LMMs. Our experimental results demonstrate that VAMBA improves accuracy by 4.3% on the challenging hour-long video understanding benchmark LVBench over prior efficient video LMMs, and maintains strong performance on a broad spectrum of long and short video understanding tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。