VeOmni让多模态大模型训练更高效，支持3D并行与快速扩展。
VeOmni: Scaling Any Modality Model Training with Model-Centric Distributed Recipe Zoo
- 用模型为中心的分布式配方解耦计算与通信，实现3D并行。
- 300亿参数多模态MoE模型在128张卡上达2800 tokens/sec/GPU。
- 新增模态只需少量代码修改，适合多模态模型研发者使用。
大型语言模型（LLMs）的进展推动了多模态理解与生成的突破，但多模态LLM训练仍面临挑战，因需处理异构模型架构，对系统设计要求高。现有框架常将模型定义与并行逻辑耦合，导致可扩展性差且工程负担重。本文提出VeOmni，一个模块化高效的训练框架，用于加速多模态LLM开发。VeOmni引入以模型为中心的分布式配方，解耦通信与计算，支持多模态LLM的高效3D并行。其灵活配置接口可无缝集成新模态，仅需少量代码变更。基于VeOmni，一个300亿参数的多模态MoE模型可在128张GPU上实现超过2,800 tokens/sec/GPU的吞吐量，并支持16万上下文长度，充分展现其在大规模多模态LLM训练中的卓越效率与可扩展性。
原文摘要 · Abstract (English)
Recent advances in large language models (LLMs) have driven impressive progress in omni-modal understanding and generation. However, training omni-modal LLMs remains a significant challenge due to the heterogeneous model architectures required to process diverse modalities, necessitating sophisticated system design for efficient large-scale training. Existing frameworks typically entangle model definition with parallel logic, incurring limited scalability and substantial engineering overhead for end-to-end omni-modal training. We present VeOmni, a modular and efficient training framework to accelerate the development of omni-modal LLMs. VeOmni introduces model-centric distributed recipes that decouples communication from computation, enabling efficient 3D parallelism on omni-modal LLMs. VeOmni also features a flexible configuration interface supporting seamless integration of new modalities with minimal code change. Using VeOmni, a omni-modal mixture-of-experts (MoE) model with 30B parameters can be trained with over 2,800 tokens/sec/GPU throughput and scale to 160K context lengths via 3D parallelism on 128 GPUs, showcasing its superior efficiency and scalability for training large omni-modal LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。