用线性RNN替代Transformer，实现高效长视频理解
AuroraLong: Bringing RNNs Back to Efficient Open-Ended Video Understanding
- 采用线性RNN处理任意长度输入，隐藏状态大小恒定
- 仅20亿参数，在多个数据集上媲美大模型性能
- 适合资源有限但需长视频理解的研究与应用
长视频理解面临计算复杂度高、内存开销大的挑战，因基于Transformer的LLM其内存和计算量随输入序列长度呈二次增长。我们提出AuroraLong，通过将多模态大模型（MLLM）中的LLM组件替换为线性RNN语言模型，以常数大小的隐藏状态处理任意长度输入序列。为进一步提升吞吐量与效率，采用按视觉标记大小升序重排的视觉标记合并策略。尽管仅有20亿参数且仅在公开数据上训练，AuroraLong在多个视频基准测试中表现与同规模、在私有数据上训练的Transformer模型相当。这表明线性RNN具备推动长视频理解平民化的潜力，显著降低计算门槛。据我们所知，这是首个在类似LLaVA的模型中使用线性RNN作为LLM主干进行开放域视频理解的工作。
原文摘要 · Abstract (English)
The challenge of long video understanding lies in its high computational complexity and prohibitive memory cost, since the memory and computation required by transformer-based LLMs scale quadratically with input sequence length. We propose AuroraLong to address this challenge by replacing the LLM component in MLLMs with a linear RNN language model that handles input sequence of arbitrary length with constant-size hidden states. To further increase throughput and efficiency, we combine visual token merge with linear RNN models by reordering the visual tokens by their sizes in ascending order. Despite having only 2B parameters and being trained exclusively on public data, AuroraLong achieves performance comparable to Transformer-based models of similar size trained on private datasets across multiple video benchmarks. This demonstrates the potential of efficient, linear RNNs to democratize long video understanding by lowering its computational entry barrier. To our best knowledge, we are the first to use a linear RNN based LLM backbone in a LLaVA-like model for open-ended video understanding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。