针对爱尔兰语等低资源语言,提出高效微调方法,让小模型媲美全量微调效果。
SemiAdapt and SemiLoRA: Efficient Domain Adaptation for Transformer-based Low-Resource Language Translation with a Case Study on Irish
- 采用半监督推理机制,仅训练少量参数实现领域自适应。
- SemiLoRA在大语料上性能超越全模型微调,且效率提升显著。
- 专为低资源语言设计,适合资源有限的研究者快速部署翻译系统。
微调广泛用于将大语言模型适配至神经机器翻译(NMT)等特定任务。然而,对含数十亿参数的多语言模型进行微调成本高昂,阻碍了爱尔兰语等低资源语言研究者参与。参数高效微调(PEFT)通过仅训练原模型的一小部分参数缓解此问题,其中低秩适配(LoRA)引入可训练的小型适配层。本文提出SemiAdapt与SemiLoRA,为半监督、推理高效的领域自适应方法,显著提升NMT整体性能。实验表明,SemiAdapt优于全领域微调;尤为关键的是,SemiLoRA使PEFT方法达到甚至超越全模型微调水平。我们还评估了按数据集的领域微调策略,并验证基于嵌入的推理方法在更大、更嘈杂语料上表现优异。本研究开发的所有爱尔兰语翻译模型均开源。这些方法旨在降低高质量领域自适应与微调的门槛,助力低资源语言研究。
原文摘要 · Abstract (English)
Fine-tuning is widely used to tailor large language models for specific tasks such as neural machine translation (NMT). However, leveraging transfer learning is computationally expensive when fine-tuning large multilingual models with billions of parameters, thus creating a barrier to entry for researchers working on low-resource domains such as Irish translation. Parameter-efficient fine-tuning (PEFT) bridges this gap by training on a fraction of the original model parameters, with the Low-Rank Adaptation (LoRA) approach introducing small, trainable adapter layers. We introduce SemiAdapt and SemiLoRA as semi-supervised inference-efficient approaches that strengthen domain adaptation and lead to improved overall performance in NMT. We demonstrate that SemiAdapt can outperform full-domain fine-tuning, while most notably, SemiLoRA can propel PEFT methods to match or even outperform full-model fine-tuning. We further evaluate domain-by-dataset fine-tuning and demonstrate that our embedding-based inference methods perform especially well on larger and noisier corpora. All Irish translation models developed in this work are released as open resources. These methods aim to make high-quality domain adaptation and fine-tuning more accessible to researchers working with low-resource languages.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。