为南非11种官方语言构建首个开源小规模解码器模型,支持多任务适配。
MzansiText and MzansiLM: An Open Corpus and Decoder-Only Language Model for South African Languages
- 构建可复现的多语言预训练语料MzansiText,训练125M参数解码器模型MzansiLM
- 单语微调在数据生成任务上达20.65 BLEU(isiXhosa),媲美十倍大的基线模型
- 适合低资源语言研究者、非洲语言技术开发者使用,提供清晰适配策略
解码器仅语言模型可通过指令微调适应多种任务,但在小规模下对低资源语言的泛化能力尚不明确。本文聚焦南非语言,首次公开一个面向全部11种官方书面语言的解码器模型,其中9种为低资源语言。我们提出MzansiText,一个可复现过滤流程的多语言预训练语料;并训练了125M参数的MzansiLM模型。评估涵盖三种适配方式:单语任务微调、多语任务微调和通用多任务指令微调。单语微调在数据到文本生成任务中表现优异,isiXhosa任务达20.65 BLEU,性能媲美参数量超十倍的编码器-解码器基线。多语微调使语义相近语言在话题分类中取得78.5%宏平均F1(isiXhosa新闻分类)。尽管模型在监督式自然语言理解与生成任务上有效,但少样本推理仍具挑战,即使更大模型也仅达随机水平。我们发布MzansiText与MzansiLM,为南非语言的小规模解码器建模提供可复现基准与适配指导。
原文摘要 · Abstract (English)
Decoder-only language models can be adapted to diverse tasks through instruction finetuning, but the extent to which this generalizes at small scale for low-resource languages remains unclear. We focus on the languages of South Africa, where we are not aware of a publicly available decoder-only model that explicitly targets all eleven official written languages, nine of which are low-resource. We introduce MzansiText, a curated multilingual pretraining corpus with a reproducible filtering pipeline, and MzansiLM, a 125M-parameter language model trained from scratch. We evaluate MzansiLM on natural language understanding and generation using three adaptation regimes: monolingual task-specific finetuning, multilingual task-specific finetuning, and general multi-task instruction finetuning. Monolingual task-specific finetuning achieves strong performance on data-to-text generation, reaching 20.65 BLEU on isiXhosa and competing with encoder-decoder baselines over ten times larger. Multilingual task-specific finetuning benefits closely related languages on topic classification, achieving 78.5% macro-F1 on isiXhosa news classification. While MzansiLM adapts effectively to supervised NLU and NLG tasks, few-shot reasoning remains challenging at this model size, with performance near chance even for much larger decoder-only models. We release MzansiText and MzansiLM to provide a reproducible decoder-only baseline and clear guidance on adaptation strategies for South African languages at small scale.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。