冻结多语言模型底层层可避免重对齐导致的性能下降。
AlignFreeze: Navigating the Impact of Realignment on the Layers of Multilingual Models Across Diverse Languages
- 通过冻结低层或高层实现重对齐，控制影响范围。
- 在35种语言中，7种语言的词性标注准确率提升超一个标准差。
- 适合需跨语言迁移且避免性能退化的场景使用。
重对齐技术常用于提升多语言模型的跨语言迁移能力，但可能在与微调源语言差异较大的语言上导致性能下降。本文提出AlignFreeze方法，在重对齐时冻结模型的下半部分或上半部分。在4项任务、3个模型、35种语言上的控制实验表明，重对齐会影响所有层，但对底层损害最严重。冻结底层可有效防止性能下降。尤其在XLM-R模型上，相比完整重对齐，AlignFreeze在7种语言中的词性标注准确率提升超过一个标准差。
原文摘要 · Abstract (English)
Realignment techniques are often employed to enhance cross-lingual transfer in multilingual language models, still, they can sometimes degrade performance in languages that differ significantly from the fine-tuned source language. This paper introduces AlignFreeze, a method that freezes either the layers' lower half or upper half during realignment. Through controlled experiments on 4 tasks, 3 models, and in 35 languages, we find that realignment affects all the layers but can be the most detrimental to the lower ones. Freezing the lower layers can prevent performance degradation. Particularly, AlignFreeze improves Part-of-Speech (PoS) tagging performances in languages where full realignment fails: with XLM-R, it provides improvements of more than one standard deviation in accuracy in seven more languages than full realignment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。