Long-VITA支持百万级标记的多模态长文本理解，且短上下文表现顶尖。
Long-VITA: Scaling Large Multi-modal Models to 1 Million Tokens with Leading Short-Context Accuracy
- 从大语言模型出发，分阶段对齐视觉与语言，训练长序列能力。
- 单机8卡下推理速度提升2倍，上下文长度扩展4倍至百万标记。
- 全开源，适合研究长序列多模态理解的开发者和学术界使用。
我们提出Long-VITA，一种适用于长上下文视觉-语言理解任务的高效多模态大模型。该模型可同时处理图像、视频（超过4000帧）和文本，输入长度达100万标记，且在短上下文多模态任务中表现卓越。采用渐进式训练策略：从大语言模型出发，依次进行视觉-语言对齐、通用知识学习及两次长序列微调。为实现无限长输入推理，引入上下文并行分布式推理与词元掩码语言建模头。训练数据仅使用公开数据集中的1700万样本，性能超越依赖内部数据的前沿模型。在单节点8张GPU上，模型实现2倍预填充加速与4倍上下文长度扩展。Long-VITA完全开源可复现，旨在为开放社区提供强有力基线，推动长上下文多模态理解发展。
原文摘要 · Abstract (English)
We introduce Long-VITA, a simple yet effective large multi-modal model for long-context visual-language understanding tasks. It is adept at concurrently processing and analyzing modalities of image, video, and text over 4K frames or 1M tokens while delivering advanced performances on short-context multi-modal tasks. We propose an effective multi-modal training schema that starts with large language models and proceeds through vision-language alignment, general knowledge learning, and two sequential stages of long-sequence fine-tuning. We further implement context-parallelism distributed inference and logits-masked language modeling head to scale Long-VITA to infinitely long inputs of images and texts during model inference. Regarding training data, Long-VITA is built on a mix of 17M samples from public datasets only and demonstrates state-of-the-art performance on various multi-modal benchmarks, compared against recent cutting-edge models with internal data. Long-VITA is fully open-source and reproducible.. By leveraging our inference designs, Long-VITA models achieve a remarkable 2x prefill speedup and 4x context length extension in a single node with 8 GPUs. We hope Long-VITA can serve as a competitive baseline and offer valuable insights for the open-source community in advancing long-context multi-modal understanding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。