让大模型一次处理近一千张图，又快又省显存。
LongLLaVA: Scaling Multi-modal LLMs to 1000 Images Efficiently via a Hybrid Architecture
- 混用Mamba和Transformer块，提升长序列处理效率
- 在A100 80GB上可处理近1000张图像，显存低、吞吐高
- 适合视频理解、高分辨率图像分析等长上下文任务
扩展多模态大模型的长上下文能力对视频理解与高分辨率图像分析至关重要。这需要系统性改进模型架构、数据构建与训练策略，尤其要应对图像数量增加导致性能下降及计算成本高的问题。本文提出一种融合Mamba与Transformer块的混合架构，设计能捕捉时空依赖的数据构建方法，并采用渐进式训练策略。所发布的LongLLaVA模型在保持高效的同时实现优异性能：在多个基准测试中表现竞争力，且具备高吞吐与低内存消耗。特别地，它可在单张A100 80GB GPU上处理接近一千张图像，展现出广泛多模态应用潜力。
原文摘要 · Abstract (English)
Expanding the long-context capabilities of Multi-modal Large Language Models~(MLLMs) is critical for advancing video understanding and high-resolution image analysis. Achieving this requires systematic improvements in model architecture, data construction, and training strategies, particularly to address challenges such as performance degradation with increasing image counts and high computational costs. In this paper, we propose a hybrid architecture that integrates Mamba and Transformer blocks, introduce data construction methods that capture both temporal and spatial dependencies, and employ a progressive training strategy. Our released model, LongLLaVA (\textbf{Long}-Context \textbf{L}arge \textbf{L}anguage \textbf{a}nd \textbf{V}ision \textbf{A}ssistant), demonstrates an effective balance between efficiency and performance. LongLLaVA achieves competitive results across various benchmarks while maintaining high throughput and low memory consumption. Notably, it can process nearly one thousand images on a single A100 80GB GPU, underscoring its potential for a wide range of multi-modal applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。