通过模仿语言切换学习,提升大模型跨语言迁移能力。
Code-Switching Curriculum Learning for Multilingual Transfer in LLMs
- 分三阶段渐进训练:词级、句级语言切换,再过渡到单一语言数据。
- 在韩语上性能显著优于传统单语持续预训练方法。
- 尤其适合低资源语言,且能减少语言资源与安全对齐的虚假关联。
大语言模型在多种任务中已接近人类水平表现,但在少数高资源语言后性能急剧下降,原因在于预训练数据分布不均。受人类二语习得中语言切换现象启发,我们提出代码切换课程学习(CSCL),以增强大模型的跨语言迁移能力。CSCL模拟人类语言学习阶段,逐步使用三类数据进行训练:1)词级语言切换,2)句级语言切换,3)单语语料库。以Qwen 2为基底模型,实验表明,相比单语持续预训练方法,CSCL在韩语上的语言迁移性能显著提升。消融实验显示,词级与句级语言切换均显著促进跨语言迁移,而课程学习进一步放大效果。研究还拓展至日语(高资源)和印尼语(低资源),并验证了在Gemma 2与Phi 3.5两个模型上的有效性。结果表明,CSCL可缓解语言资源与安全对齐之间的虚假相关性,构建更公平、高效的多语言迁移框架。尤其在缺乏高质量单语语料的低资源场景下表现优异。
原文摘要 · Abstract (English)
Large language models (LLMs) now exhibit near human-level performance in various tasks, but their performance drops drastically after a handful of high-resource languages due to the imbalance in pre-training data. Inspired by the human process of second language acquisition, particularly code-switching$\unicode{x2014}$the practice of language alternation in a conversation$\unicode{x2014}$we propose code-switching curriculum learning (CSCL) to enhance cross-lingual transfer for LLMs. CSCL mimics the stages of human language learning by progressively training models with a curriculum consisting of 1) token-level code-switching, 2) sentence-level code-switching, and 3) monolingual corpora. Using Qwen 2 as our underlying model, we demonstrate the efficacy of the CSCL in improving language transfer to Korean, achieving significant performance gains compared to monolingual continual pre-training methods. Ablation studies reveal that both token- and sentence-level code-switching significantly enhance cross-lingual transfer and that curriculum learning amplifies these effects. We also extend our findings into various languages, including Japanese (high-resource) and Indonesian (low-resource), and using two additional models (Gemma 2 and Phi 3.5). We further show that CSCL mitigates spurious correlations between language resources and safety alignment, presenting a robust, efficient framework for more equitable language transfer in LLMs. We observe that CSCL is effective for low-resource settings where high-quality, monolingual corpora for language transfer are hardly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。