arXiv:2603.20732cs.CL2026-03中稿 · LREC 2026

为南非11种官方语言构建首个开源小规模解码器模型,支持多任务适配。

MzansiText and MzansiLM: An Open Corpus and Decoder-Only Language Model for South African Languages

  • 构建可复现的多语言预训练语料MzansiText,训练125M参数解码器模型MzansiLM
  • 单语微调在数据生成任务上达20.65 BLEU(isiXhosa),媲美十倍大的基线模型
  • 适合低资源语言研究者、非洲语言技术开发者使用,提供清晰适配策略

解码器仅语言模型可通过指令微调适应多种任务,但在小规模下对低资源语言的泛化能力尚不明确。本文聚焦南非语言,首次公开一个面向全部11种官方书面语言的解码器模型,其中9种为低资源语言。我们提出MzansiText,一个可复现过滤流程的多语言预训练语料;并训练了125M参数的MzansiLM模型。评估涵盖三种适配方式:单语任务微调、多语任务微调和通用多任务指令微调。单语微调在数据到文本生成任务中表现优异,isiXhosa任务达20.65 BLEU,性能媲美参数量超十倍的编码器-解码器基线。多语微调使语义相近语言在话题分类中取得78.5%宏平均F1(isiXhosa新闻分类)。尽管模型在监督式自然语言理解与生成任务上有效,但少样本推理仍具挑战,即使更大模型也仅达随机水平。我们发布MzansiText与MzansiLM,为南非语言的小规模解码器建模提供可复现基准与适配指导。

原文摘要 · Abstract (English)

Decoder-only language models can be adapted to diverse tasks through instruction finetuning, but the extent to which this generalizes at small scale for low-resource languages remains unclear. We focus on the languages of South Africa, where we are not aware of a publicly available decoder-only model that explicitly targets all eleven official written languages, nine of which are low-resource. We introduce MzansiText, a curated multilingual pretraining corpus with a reproducible filtering pipeline, and MzansiLM, a 125M-parameter language model trained from scratch. We evaluate MzansiLM on natural language understanding and generation using three adaptation regimes: monolingual task-specific finetuning, multilingual task-specific finetuning, and general multi-task instruction finetuning. Monolingual task-specific finetuning achieves strong performance on data-to-text generation, reaching 20.65 BLEU on isiXhosa and competing with encoder-decoder baselines over ten times larger. Multilingual task-specific finetuning benefits closely related languages on topic classification, achieving 78.5% macro-F1 on isiXhosa news classification. While MzansiLM adapts effectively to supervised NLU and NLG tasks, few-shot reasoning remains challenging at this model size, with performance near chance even for much larger decoder-only models. We release MzansiText and MzansiLM to provide a reproducible decoder-only baseline and clear guidance on adaptation strategies for South African languages at small scale.

语言模型低资源语言南非语言开源数据

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。