发现多语言翻译微调可仅用极小参数子空间，提升效率并避免语言冲突。
Exploring Intrinsic Language-specific Subspaces in Fine-tuning Multilingual Neural Machine Translation
- 为每种语言设计专属低秩适配器，只更新其内在语言特定子空间。
- 在FLORES-101上实现最高2.25的spBLEU提升，参数量降至0.4%~1.6%。
- 适合需要高效多语言模型微调的研究者和工业应用开发者。
多语言神经机器翻译模型可同时微调数百种语言，但全参数微调效率低，易引发语言间负向干扰。本文证明，每种语言的微调实际上发生在其内在的语言特定子空间中，仅需极少量参数。为此，提出语言特定LoRA，用于隔离这些内在子空间。进一步引入架构学习技术与渐进式剪枝策略，系统探索最优设置与最小必要子空间，实现轻量且高效的微调流程。在FLORES-101的12语种和30语种子集上的实验表明，该方法不仅比全参数微调最高提升2.25 spBLEU，还将高、中资源语言的可训练参数压缩至0.4%，低资源语言为1.6%。
原文摘要 · Abstract (English)
Multilingual neural machine translation models support fine-tuning hundreds of languages simultaneously. However, fine-tuning on full parameters solely is inefficient potentially leading to negative interactions among languages. In this work, we demonstrate that the fine-tuning for a language occurs in its intrinsic language-specific subspace with a tiny fraction of entire parameters. Thus, we propose language-specific LoRA to isolate intrinsic language-specific subspaces. Furthermore, we propose architecture learning techniques and introduce a gradual pruning schedule during fine-tuning to exhaustively explore the optimal setting and the minimal intrinsic subspaces for each language, resulting in a lightweight yet effective fine-tuning procedure. The experimental results on a 12-language subset and a 30-language subset of FLORES-101 show that our methods not only outperform full-parameter fine-tuning up to 2.25 spBLEU scores but also reduce trainable parameters to $0.4\%$ for high and medium-resource languages and $1.6\%$ for low-resource ones.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。