通过优化参数位置，提升小模型在冰岛语上的生成能力
Train More Parameters But Mind Their Placement: Insights into Language Adaptation with PEFT
- 在前馈层和瓶颈适配器中增加可训练参数，效果更优
- 参数增多显著改善语言适配性能，但全层微调易影响长上下文处理
- 仅微调最后几层可兼顾语言特化与长文本理解
小规模大语言模型在中等资源语言中仍面临显著挑战，尤其在语言特定知识获取方面，单纯依赖机器翻译数据难以解决。本文以冰岛语为例，通过无结构文本语料对LLM进行专业化适配，重点避免微调过程干扰模型对长上下文的处理能力。通过多种参数高效微调（PEFT）方法的消融实验发现，增加可训练参数数量能带来更好、更稳健的语言适配效果。在前馈层和瓶颈适配器中使用足够参数的LoRA与适配器表现优异，而前缀提示和(IA)3则不适用。尽管0-shot摘要任务中改进一致，部分模型在长上下文任务中表现下降，该问题可通过仅微调最后几层得以缓解。
原文摘要 · Abstract (English)
Smaller LLMs still face significant challenges even in medium-resourced languages, particularly when it comes to language-specific knowledge -- a problem not easily resolved with machine-translated data. In this case study on Icelandic, we aim to enhance the generation performance of an LLM by specialising it using unstructured text corpora. A key focus is on preventing interference with the models' capabilities of handling longer context during this adaptation. Through ablation studies using various parameter-efficient fine-tuning (PEFT) methods and setups, we find that increasing the number of trainable parameters leads to better and more robust language adaptation. LoRAs placed in the feed-forward layers and bottleneck adapters show promising results with sufficient parameters, while prefix tuning and (IA)3 are not suitable. Although improvements are consistent in 0-shot summarisation, some adapted models struggle with longer context lengths, an issue that can be mitigated by adapting only the final layers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。