用少量数据让大模型跨语言迁移，同时不丢英语能力
Cross-Lingual Optimization for Language Transfer in Large Language Models
- 用英文微调数据+翻译模型实现跨语言迁移
- 低资源语言仅需3200样本，效果超传统方法6400样本
- 适合资源少的语言迁移，尤其对数据敏感场景
将大语言模型适配到其他语言通常采用监督微调（SFT），但常过度侧重英语表现，尤其在数据受限时更明显。为此，我们提出跨语言优化（CLO），可在保持英语能力的同时高效迁移到目标语言。CLO利用公开的英文微调数据和翻译模型实现跨语言转移。我们在五个模型上对六种语言进行了实验，涵盖不同资源水平。结果表明，CLO在提升目标语言能力的同时，始终优于SFT。尤其在低资源语言中，仅用3200样本的CLO就超越了使用6400样本的SFT，证明其以更少数据获得更好效果。此外，SFT在中低资源语言中对数据量高度敏感，而CLO则表现稳定。综合分析揭示了SFT的局限性，并在CLO中引入额外训练策略以提升效率。
原文摘要 · Abstract (English)
Adapting large language models to other languages typically employs supervised fine-tuning (SFT) as a standard approach. However, it often suffers from an overemphasis on English performance, a phenomenon that is especially pronounced in data-constrained environments. To overcome these challenges, we propose \textbf{Cross-Lingual Optimization (CLO)} that efficiently transfers an English-centric LLM to a target language while preserving its English capabilities. CLO utilizes publicly available English SFT data and a translation model to enable cross-lingual transfer. We conduct experiments using five models on six languages, each possessing varying levels of resource. Our results show that CLO consistently outperforms SFT in both acquiring target language proficiency and maintaining English performance. Remarkably, in low-resource languages, CLO with only 3,200 samples surpasses SFT with 6,400 samples, demonstrating that CLO can achieve better performance with less data. Furthermore, we find that SFT is particularly sensitive to data quantity in medium and low-resource languages, whereas CLO remains robust. Our comprehensive analysis emphasizes the limitations of SFT and incorporates additional training strategies in CLO to enhance efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。