用低资源突厥语巴什基尔语测试高效微调方法，发现QLoRA在70亿模型上效果接近全量微调。
Adapting Large Language Models to a Low-Resource Agglutinative Language: A Comparative Study of LoRA and QLoRA for Bashkir
- 对比LoRA与QLoRA在低资源黏着语上的表现，采用多模型多随机种子验证可靠性。
- QLoRA在Mistral-7B和Phi-2上仅需40倍少参数即达3.79-3.81困惑度，接近全微调最佳3.34。
- 模型选择和分词器影响大，部分架构如DeepSeek-7B用低秩适配时性能暴跌至129.55。
本文对比了参数高效微调（PEFT）方法，包括LoRA与QLoRA，在适应低资源黏着语巴什基尔语任务中的表现。实验基于包含71,000篇文档（4690万词元）的巴什基尔文本语料库，使用多种架构模型：DistilGPT2、GPT-2（基础版、中等版）、Phi-2、Qwen2.5-7B、DeepSeek-7B 和 Mistral-7B。为提升结果可靠性，每种配置均以三个不同随机种子训练。测试集最低困惑度由全量微调的GPT-2中等版取得（3.34）。而应用于Mistral-7B（3.79）和Phi-2（3.81）的QLoRA，仅需超过40倍少的可训练参数即可达到相近质量。然而，某些架构在使用PEFT时出现显著性能下降（如DeepSeek-7B在秩为8时，困惑度达129.55），表明结果对基础模型及其分词器高度敏感。此外，基于巴什基尔提示的生成文本定性分析显示，虽全微调模型困惑度最低，但常在输出中切换至英文；而QLoRA微调模型则更稳定地生成单语巴什基尔文本。结果表明，70亿级模型上的QLoRA在质量与计算成本间提供有效权衡。为确保可复现性，论文将公开数据、代码及训练好的适配器。
原文摘要 · Abstract (English)
This paper presents a comparative study of parameter-efficient fine-tuning (PEFT) methods, including LoRA and QLoRA, applied to the task of adapting large language models to the Bashkir language, a low-resource agglutinative language of the Turkic family. Experimental evaluation is conducted on a Bashkir text corpus of 71k documents (46.9M tokens) using models of various architectures: DistilGPT2, GPT-2 (base, medium), Phi-2, Qwen2.5-7B, DeepSeek-7B, and Mistral-7B. To improve the reliability of results, each configuration was trained with three different random seeds. The lowest perplexity on the test set was obtained for GPT-2 medium with full fine-tuning (3.34). Meanwhile, QLoRA applied to Mistral-7B (3.79) and Phi-2 (3.81) achieved comparable quality with over 40 times fewer trainable parameters. However, we also observed cases of significant quality degradation when using PEFT for certain architectures (e.g., DeepSeek-7B with rank 8, perplexity = 129.55), indicating that the outcome depends critically on the choice of the base model and its tokenizer. Additionally, a qualitative analysis of generated texts based on Bashkir prompts revealed that models with the best perplexity do not necessarily produce the most coherent outputs: QLoRA-tuned models generated monolingual Bashkir continuations, whereas the fully fine-tuned model with the lowest perplexity frequently switched to English. The results suggest that QLoRA on 7B-scale models offers an effective compromise between quality and computational cost for Bashkir. To ensure reproducibility, open data, code, and trained adapters will be released upon acceptance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。