用专家模型高效扩展多语言大模型，不需大量数据就能保持原有能力。
A Data-Efficient Path to Multilingual LLMs: Language Expansion via Post-training PARAM$Δ$ Integration into Upcycled MoE
- 将稠密模型升级为分语言专家的混合专家架构，按语言分配不同专家。
- 通过嫁接预训练后的参数增量，绕过耗时对齐阶段，实现语言扩展。
- 在不牺牲原模型能力的前提下，显著提升新增语言性能，适合多语言部署场景。
将大语言模型扩展至新语言成本高昂，需大量持续预训练（CPT）和数据密集型对齐。现有无数据融合技术虽能绕过对齐，但面临核心矛盾：缓解参数冲突以保留原能力会削弱新语言学习，反之亦然。为此，我们提出 extmethod，将稠密模型改造为混合专家（MoE）架构，为不同语言分配专属专家。通过将经过后训练的专家扩展参数增量（Δpost）嫁接到增强版基模型上，实现对齐能力迁移，跳过复杂对齐过程。实验表明，即使与同类模型在浮点运算量（FLOPs）或参数量相当的情况下， extmethod仍显著提升新增语言表现，并有效保留原有能力。该方法可广泛适配多种模型及后训练参数增量。
原文摘要 · Abstract (English)
Expanding Large Language Models~(LLMs) to new languages is a costly endeavor, demanding extensive Continued Pre-Training~(CPT) and data-intensive alignment. While recent data-free merging techniques attempt to bypass alignment by fusing a multilingual CPT-enhanced model with its instruct counterpart, they are plagued by a critical trade-off: mitigating parameter conflicts to preserve original abilities inevitably dilutes new language acquisition, and vice-versa. To resolve this conflict, we introduce \method, which upcycles a dense model into a Mixture-of-Experts~(MoE) architecture, allocating different experts to different languages. Alignment ability is then transferred by grafting a MoE-expanded parameter delta~($Δ_{\text{post}}$) to the CPT-enhanced base model, bypassing the complex alignment phase. Experiments demonstrate \method's superiority even against baselines with similar FLOPs or number of parameters; it improves performance on expanded languages while effectively preserving original capabilities. We further show our approach is highly applicable across different models and Post-training deltas.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。