通过语言特定模型合并，显著提升多语言大模型训练效率并降低维护成本。
Improving Training Efficiency and Reducing Maintenance Costs via Language Specific Model Merging
- 按语言分别训练后合并模型，避免全量重训。
- 初始训练时间减少50%，语言更新成本降低60%以上。
- 适用于工业和学术场景，兼顾效率与性能。
为任务定制的多语言大语言模型微调需在包含所有目标语言的数据集上训练。当需要新增数据或支持新语言时，通常需重新训练整个模型，计算开销大且维护困难。近期研究显示模型合并可提升质量，但其效率尚未被系统评估。本文首次从效率角度分析该策略，在三个独立任务上验证：该方法将初始训练时间缩短最高达50%；仅更新单个语言并重新合并，相比全模型重训，训练成本降低超过60%。实验涵盖公开与企业私有数据集，证明该方法在工业与学术场景中均有效。
原文摘要 · Abstract (English)
Fine-tuning a task-specific multilingual large language model (LLM) involves training the model on a multilingual dataset with examples in all the required languages. Updating one or more supported languages with additional data or adding support for a new language involves retraining the model, which can be computationally inefficient and creates a severe maintenance bottleneck. Recent research on merging multilingual multitask models has shown promise in terms of improved quality, but its computational and maintenance efficiency remains unstudied. In this work, we provide the first focused analysis of this merging strategy from an efficiency perspective, evaluating it across three independent tasks. We demonstrate significant efficiency gains while maintaining parity in terms of quality: this merging approach reduces the initial training time by up to 50\%. We also demonstrate that updating an individual language and re-merging as part of model maintenance reduces training costs by more than 60\%, compared to re-training the full multilingual model. We show this on both public and proprietary industry datasets confirming that the approach works well for industrial use cases in addition to academic settings already studied in previous work.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。