对比多种微调方法在德语族语言上的表现，发现不同语言和任务适用不同策略。
How to Tune a Multilingual Encoder Model for Germanic Languages: A Study of PEFT, Full Fine-Tuning, and Language Adapters
- 比较全量微调、LoRA与Pfeiffer适配器在三种语言上的效果
- 德国语用PEFT更优，但瑞典语和冰岛语结果不一致
- 问答任务适合PEFT，命名实体识别则倾向全量微调
本文研究多语言编码模型mDeBERTa在德语、瑞典语和冰岛语三种德语族语言上的最优使用方式，这些语言在mDeBERTa预训练数据中的资源丰富程度和数据质量各异。对比全量微调与参数高效微调（PEFT）方法（包括LoRA和Pfeiffer瓶颈适配器），发现对于高资源语言德国语，PEFT表现更优；而瑞典语和冰岛语的结果则不够稳定。此外，任务类型影响效果：在问答任务中PEFT更具优势，而在命名实体识别任务中全量微调更佳。受模块化适配器研究启发，进一步评估在非结构化文本上训练的PEFT模块是否有效，结果表明该方法并无提升作用。
原文摘要 · Abstract (English)
This paper investigates the optimal use of the multilingual encoder model mDeBERTa for tasks in three Germanic languages -- German, Swedish, and Icelandic -- representing varying levels of presence and likely data quality in mDeBERTas pre-training data. We compare full fine-tuning with the parameter-efficient fine-tuning (PEFT) methods LoRA and Pfeiffer bottleneck adapters, finding that PEFT is more effective for the higher-resource language, German. However, results for Swedish and Icelandic are less consistent. We also observe differences between tasks: While PEFT tends to work better for question answering, full fine-tuning is preferable for named entity recognition. Inspired by previous research on modular approaches that combine task and language adapters, we evaluate the impact of adding PEFT modules trained on unstructured text, finding that this approach is not beneficial.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。