首个针对塔吉克语生成的高效微调基准,验证了QLoRA在低资源语言上的优势。
Benchmarking Parameter-Efficient Fine-Tuning of Large Language Models for Low-Resource Tajik Text Generation with the Tajik Web Corpus
- 用QLoRA对Mistral 7B进行低秩微调,实现高质量塔吉克语生成
- QLoRA r=8时困惑度5.11,仅需14.21GB显存和33分钟训练
- 避免小模型全微调,防止灾难性遗忘,适合低资源语言研究者
我们发布了包含319,000篇文档、11.1亿字符的塔吉克语网络语料库(Tajik Web Corpus),并基于此对九种架构的十七种配置在塔吉克语提示续写任务上进行了参数高效微调(PEFT)基准测试。评估策略包括全量微调、LoRA和QLoRA(秩8与16)。由于不同模型分词器导致困惑度不可直接比较,质量通过各模型家族内部困惑度对比结合母语者定性分析评估。结果显示,使用QLoRA秩8的Mistral 7B表现最佳:困惑度5.11(同家族内),生成文本连贯且符合塔吉克语规范,仅消耗14.21GB GPU内存,训练时间约33分钟。秩提升至16后,困惑度仅降至5.03(成对p>0.05),但多耗约1GB内存。小规模GPT-2全微调虽数值困惑度更低,但引发灾难性遗忘(输出英文或乱码);而QLoRA保留多语言预训练知识,生成有意义塔吉克语。编码器模型表现最差(困惑度约59),证实其不适用于自回归生成。本研究为首个针对塔吉克语生成的系统性PEFT基准,建议采用Mistral 7B+QLoRA r=8,避免小模型全微调,并使用所发布的语料库与基准。
原文摘要 · Abstract (English)
We release the Tajik Web Corpus (319k docs, 1.11B chars) and benchmark generative LLMs on prompt continuation in Tajik, a low-resource Cyrillic-script language. Seventeen configurations across nine architectures are evaluated under three fine-tuning strategies: full fine-tuning, LoRA, and QLoRA (ranks 8 and 16). Because perplexity is not directly comparable across model families with different tokenizers, generation quality is assessed through perplexity interpreted within each family, complemented by qualitative analysis performed by a native Tajik speaker. Computational cost is measured via GPU memory and training time. The best quality-cost trade-off is achieved by Mistral 7B with QLoRA rank 8: perplexity 5.11 (within its tokenizer family), coherent Tajik output confirmed by the native speaker, 14.21 GB GPU memory, and approximately 33 minutes of training. Increasing the rank to 16 yields a negligible improvement for Mistral (perplexity 5.03, pairwise p > 0.05) while consuming about 1 GB more memory. Full fine-tuning of small GPT-2 models obtains lower numeric perplexity but leads to catastrophic forgetting (English or gibberish output); in contrast, QLoRA preserves multilingual pretrained knowledge and generates meaningful Tajik text. Encoder-only models perform worst (perplexity approximately 59), confirming their unsuitability for autoregressive generation. To our knowledge, this is the first systematic PEFT benchmark for Tajik text generation. Practical recommendations include using Mistral 7B with QLoRA r=8, avoiding full fine-tuning of small GPT-2 models, and adopting the released corpus and benchmark.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。