在昇腾芯片上实现高效多模态大模型训练,数据效率提升显著。
MindVL: Towards Efficient and Effective Training of Multimodal Large Language Models on Ascend NPUs
- 基于昇腾NPU构建高效训练框架,支持稠密与专家混合模型训练。
- 仅用10%数据达到7B模型性能,3%数据匹配72B模型表现。
- 开源数据配方与权重平均策略,助力复现与性能优化。
我们提出MindVL,一个在昇腾NPU上训练的多模态大语言模型。当前先进多模态大模型训练受限于少数硬件平台,且依赖大量未公开的数据配方,阻碍了可复现性与开放研究。为改变昇腾硬件不适用于高效全阶段训练的固有认知,我们引入MindSpeed-MLLM,一个高效率训练框架,支持在昇腾硬件上稳定、高性能地训练大规模稠密与专家混合(MoE)模型。在此基础上,我们系统化并开源了各训练阶段的数据生成方法与混合策略。进一步,我们提出MindVL,一个端到端在昇腾NPU上训练的数据高效多模态大语言模型。实验发现,对不同序列长度训练得到的检查点权重取平均,在结合测试时分辨率搜索时尤为有效,能带来进一步性能提升。结果显示:MindVL-8B仅用10%的训练数据即达到Qwen2.5VL-7B的性能;而我们的MoE模型MindVL-671B-A37B仅需Qwen2.5VL训练数据的3%,即可匹配其72B模型性能,并与当前领先多模态MoE模型表现相当。本工作为社区提供了有价值的硬件替代方案、开源数据配方及有效的性能增强技术。
原文摘要 · Abstract (English)
We propose MindVL, a multimodal large language model (MLLMs) trained on Ascend NPUs. The training of state-of-the-art MLLMs is often confined to a limited set of hardware platforms and relies heavily on massive, undisclosed data recipes, which hinders reproducibility and open research. To change the common perception that Ascend hardware is unsuitable for efficient full-stage MLLM training, we introduce MindSpeed-MLLM, a highly efficient training framework that supports stable and high-performance training of large-scale Dense and Mixture-of-Experts (MoE) models on Ascend hardware. Based on this, we provide a systematic and open description of the data production methods and mixing strategies for all training stages. Furthermore, we present MindVL, a data-efficient multimodal large language model trained end-to-end on Ascend NPUs. In addition, we find that averaging weights from checkpoints trained with different sequence lengths is particularly effective and yields further gains when combined with test-time resolution search. Our experiments demonstrate superior data efficiency: MindVL-8B matches the performance of Qwen2.5VL-7B using only 10\% of its training data, while our MoE model, MindVL-671B-A37B, matches Qwen2.5VL-72B using only 3\% of the Qwen2.5VL training data, and achieves comparable performance with other leading multimodal MoE models. Our work provides the community with a valuable hardware alternative, open data recipes, and effective performance-enhancing techniques.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。