用小语种数据训练生成模型，发现遮蔽方法生成文本质量略高
Pre-training a Transformer-Based Generative Model Using a Small Sepedi Dataset
- 用塞佩迪语数据集预训练变压器模型，对比遮蔽与非遮蔽方法
- 非遮蔽模型在损失和困惑度上表现更好，但遮蔽模型生成文本质量略优
- 适合低资源语言生成研究者参考，尤其关注文本质量提升
由于低资源语言数据稀缺，其语言模型发展缓慢。当前预训练语言模型在自然语言处理中广受欢迎，尤其适用于低资源语言的领域特定模型开发。本研究探索在文本生成任务中使用遮蔽技术对语言模型训练的影响。我们构建了两个新数据集：来自南非资源的塞佩迪语单语数据集（SepMono）和来自广播新闻领域的塞佩迪语广播新闻数据集（SepNews）。使用SepMono数据集，通过遮蔽与非遮蔽预训练技术对基于Transformer的模型进行训练并比较性能；SepNews数据集则用于微调。结果表明，非遮蔽模型在验证损失和困惑度上表现更优；然而，通过BLEU评分评估生成文本质量时，遮蔽模型表现略好。
原文摘要 · Abstract (English)
Due to the scarcity of data in low-resourced languages, the development of language models for these languages has been very slow. Currently, pre-trained language models have gained popularity in natural language processing, especially, in developing domain-specific models for low-resourced languages. In this study, we experiment with the impact of using occlusion-based techniques when training a language model for a text generation task. We curate 2 new datasets, the Sepedi monolingual (SepMono) dataset from several South African resources and the Sepedi radio news (SepNews) dataset from the radio news domain. We use the SepMono dataset to pre-train transformer-based models using the occlusion and non-occlusion pre-training techniques and compare performance. The SepNews dataset is specifically used for fine-tuning. Our results show that the non-occlusion models perform better compared to the occlusion-based models when measuring validation loss and perplexity. However, analysis of the generated text using the BLEU score metric, which measures the quality of the generated text, shows a slightly higher BLEU score for the occlusion-based models compared to the non-occlusion models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。