用合成指令让大模型学巴斯克语，效果接近顶级模型。
Instructing Large Language Models for Low-Resource Languages: A Systematic Study for Basque
- 用已有多语言模型生成合成指令，替代真实标注数据。
- 使用指令微调过的基座模型，性能显著优于基础模型。
- 仅用巴斯克语语料和合成指令，逼近更大模型表现。
在低资源语言场景下，大规模指令数据集难以获取。本文针对巴斯克语，系统研究了在仅有目标语言语料、开放权重的多语言基座与指令微调模型、以及从指令模型中合成的指令的前提下，如何高效训练语言模型。通过在1680名参与者上的基准测试与人工偏好评估，实验表明：目标语言语料至关重要，合成指令可生成鲁棒模型，尤其使用指令微调过的模型作为基座时表现更优。当采用Llama 3.1 Instruct 70B作为基座时，无需任何巴斯克语指令，模型性能已接近更大规模的前沿模型。代码、模型、指令数据集及人工偏好数据均公开，支持未来研究复现。
原文摘要 · Abstract (English)
Instructing language models with user intent requires large instruction datasets, which are only available for a limited set of languages. In this paper, we explore alternatives to conventional instruction adaptation pipelines in low-resource scenarios. We assume a realistic scenario for low-resource languages, where only the following are available: corpora in the target language, existing open-weight multilingual base and instructed backbone LLMs, and synthetically generated instructions sampled from the instructed backbone. We present a comprehensive set of experiments for Basque that systematically study different combinations of these components evaluated on benchmarks and human preferences from 1,680 participants. Our conclusions show that target language corpora are essential, with synthetic instructions yielding robust models, and, most importantly, that using as backbone an instruction-tuned model outperforms using a base non-instructed model. Scaling up to Llama 3.1 Instruct 70B as backbone, our model comes near frontier models of much larger sizes for Basque, without using any Basque instructions. We release code, models, instruction datasets, and human preferences to support full reproducibility in future research on low-resource language adaptation. https://github.com/hitz-zentroa/latxa-instruct
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。