arXiv:2504.01540cs.CL2025-04被引 1

为丹麦语设计基于词素的分词器,提升大模型对语言结构的理解。

From Smør-re-brød to Subwords: Training LLMs on Danish, One Morpheme at a Time

  • 利用丹麦语词素标注数据,训练半监督分词模型生成定制分词器。
  • 自研分词器在词素分割上达F1 58.84,远超丹麦语BPE的39.28。
  • 使用新分词器训练的模型在下游任务中表现更优,适合小语种NLP研究者。

当前最优的Transformer语言模型普遍采用子词分词技术(如字节对编码,BPE),但这类方法常忽略语言学中的形态分割原则。本研究利用标注的丹麦语词素数据集,训练了一个半监督词素分割模型,进而开发出针对丹麦语形态结构优化的分词器。我们评估了四种分词器,包括两种自研的形态分词器,并在 extit{CerebrasGPT-111M}和 extit{LLaMA-3.2 1B}两个生成式Transformer模型上进行训练与测试。结果表明,自研分词器在词素分割任务中达到F1分数58.84,显著优于丹麦语BPE分词器的39.28。在下游任务中,使用形态分词器训练的模型在各项指标上均优于使用BPE分词器的模型。该研究证明,将丹麦语形态结构融入分词器设计,能有效提升生成式模型在丹麦语上的性能。

原文摘要 · Abstract (English)

The best performing transformer-based language models use subword tokenization techniques, such as Byte-Pair-Encoding (BPE). However, these approaches often overlook linguistic principles, such as morphological segmentation, which we believe is fundamental for understanding language-specific word structure. In this study, we leverage an annotated Danish morphological dataset to train a semisupervised model for morphological segmentation, enabling the development of tokenizers optimized for Danish morphology. We evaluate four distinct tokenizers, including two custom morphological tokenizers, by analyzing their performance in morphologically segmenting Danish words. Additionally, we train two generative transformer models, \textit{CerebrasGPT-111M} and \textit{LLaMA-3.2 1B}, using these tokenizers and evaluate their downstream performance. Our findings reveal that our custom-developed tokenizers substantially enhance morphological segmentation, achieving an F1 score of 58.84, compared to 39.28 achieved by a Danish BPE tokenizer. In downstream tasks, models trained with our morphological tokenizers outperform those using BPE tokenizers across different evaluation metrics. These results highlight that incorporating Danish morphological segmentation strategies into tokenizers leads to improved performance in generative transformer models on Danish language

语言模型分词器丹麦语形态学

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。