对比多种LoRA变体，发现基础LoRA在多语言指令微调中已足够。
Which LoRA? An Empirical Study on the Effectiveness of LoRA Techniques During Multilingual Instruction Tuning
- 在多语言数据上测试五种LoRA变体，包括基础版与复杂变体。
- 不同变体在跨语言迁移与知识保留上无显著差异。
- 层间语言表征相似，说明结构创新未必提升多语适应能力。
我们研究了在多语言指令微调中，常见的LoRA变体是否优于基础LoRA。在两个数据集、多种目标语言上的实验表明，使用更复杂的LoRA变体相比基础LoRA，并未在平衡跨语言迁移与知识保留方面带来显著优势。对隐藏表示的分析显示，采用不同LoRA技术微调的大语言模型，其层间语言表征仍高度相似，暗示LoRA架构的创新性可能无法转化为更好的跨语言适应能力。
原文摘要 · Abstract (English)
We investigate whether commonly available LoRA variants have an advantage over basic LoRA in multilingual instruction tuning. Experiments involving LoRA and four other variants on two datasets across diverse target languages show that there is no significant advantage in using more complex LoRA variants instead of basic LoRA, with respect to balancing cross-lingual transfer and knowledge retention. An analysis of hidden embeddings reveal that layer-wise language representation remains largely similar across LLMs fine-tuned with different LoRA techniques, suggesting that architectural novelty of LoRA techniques may not translate into better cross-lingual adaptation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。