用线性复杂度模型实现高效视频字幕生成，速度提升3倍
ABMAMBA: Multimodal Large Language Model with Aligned Hierarchical Bidirectional Scan for Efficient Video Captioning
- 采用分层双向扫描机制替代传统注意力
- 在VATEX和MSR-VTT上性能接近主流模型
- 适合需要高速处理长视频的场景
本研究聚焦于开放型多模态大语言模型（MLLM）在视频字幕生成中的应用。视觉序列理解因复杂的时序依赖和长序列长度而困难，现有基于Transformer的方法注意力机制随序列长度呈平方增长，计算开销巨大。为此，我们提出对齐分层双向扫描Mamba（ABMamba），一种具有线性计算复杂度的全开源MLLM，可高效处理长视频序列。ABMamba以深度状态空间模型为语言主干，取代昂贵的二次注意力机制，并引入新型对齐分层双向扫描模块，跨多时间分辨率处理视频。在标准视频字幕基准如VATEX和MSR-VTT上，ABMamba性能与典型MLLM相当，同时吞吐量提升约三倍。
原文摘要 · Abstract (English)
In this study, we focus on video captioning by fully open multimodal large language models (MLLMs). The comprehension of visual sequences is challenging because of their intricate temporal dependencies and substantial sequence length. The core attention mechanisms of existing Transformer-based approaches scale quadratically with the sequence length, making them computationally prohibitive. To address these limitations, we propose Aligned Hierarchical Bidirectional Scan Mamba (ABMamba), a fully open MLLM with linear computational complexity that enables the scalable processing of video sequences. ABMamba extends Deep State Space Models as its language backbone, replacing the costly quadratic attention mechanisms, and employs a novel Aligned Hierarchical Bidirectional Scan module that processes videos across multiple temporal resolutions. On standard video captioning benchmarks such as VATEX and MSR-VTT, ABMamba demonstrates competitive performance compared to typical MLLMs while achieving approximately three times higher throughput.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。