arXiv:2603.28263cs.CLcs.AI2026-03中稿 · the 15th edition o…

通过模型合并,让大模型学会用小语种说话

Merge and Conquer: Instructing Multilingual Models by Adding Target Language Weights

  • 用目标语言的基座模型与指令微调模型合并,替代传统微调
  • 在4种伊比利亚语言上实现有效指令响应,性能接近传统方法
  • 适合资源匮乏语言,大幅降低算力需求,易扩展多语言能力

大型语言模型仍以英语为主,低资源语言表现有限。现有适应方法如持续预训练需大量算力,而指令微调又依赖高质量指令数据,对低资源语言社区难以获取。在此背景下,模型合并提供轻量级替代方案,但其在低资源场景下的潜力尚未系统探索。本文研究是否可通过将指令微调模型与特定语言的基座模型合并,实现语言知识迁移,从而避免每次需要新指令或重复微调。在四种伊比利亚语言(巴斯克语、加泰罗尼亚语、加利西亚语、西班牙语)和两种模型族上的实验表明,合并可有效实现新语言的指令遵循行为,甚至通过组合多个语言模型支持多语言能力。结果表明,模型合并是低资源语言的有效高效替代方法,在保持竞争力的同时显著降低计算成本。

原文摘要 · Abstract (English)

Large Language Models (LLMs) remain heavily centered on English, with limited performance in low-resource languages. Existing adaptation approaches, such as continual pre-training, demand significant computational resources. In the case of instructed models, high-quality instruction data is also required, both of which are often inaccessible for low-resource language communities. Under these constraints, model merging offers a lightweight alternative, but its potential in low-resource contexts has not been systematically explored. In this work, we explore whether it is possible to transfer language knowledge to an instruction-tuned LLM by merging it with a language-specific base model, thereby eliminating the need of language-specific instructions and repeated fine-tuning processes whenever stronger instructed variants become available. Through experiments covering four Iberian languages (Basque, Catalan, Galician, and Spanish) and two model families, we show that merging enables effective instruction following behavior in new languages and even supports multilingual capability through the combination of multiple language-specific models. Our results indicate that model merging is a viable and efficient alternative to traditional adaptation methods for low-resource languages, achieving competitive performance while greatly reducing computational cost.

模型合并多语言低资源语言指令微调

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。