arXiv:2509.15255cs.CL2025-09

对比三种分词算法,发现SentencePiece最适合不丹语杜仲喀。

Comparative Analysis of Tokenization Algorithms for Low-Resource Language Dzongkha

  • 用BPE、WordPiece和SentencePiece评估杜仲喀语分词效果
  • SentencePiece在子词密度等指标上表现最优,执行效率高
  • 为低资源语言构建大模型提供关键分词支持

大型语言模型(LLMs)发展迅速,分词器是自然语言处理中的关键组件。分词器将文本拆分为模型可处理的词汇单元,同时保持语义与结构的准确性。有效的分词器能提升模型对上下文的理解能力,从而改善翻译、分类、情感分析和文本生成等下游任务表现。现有预训练分词器多针对英语等高资源语言,对低资源语言如杜仲喀语(约70万使用者)效果不佳。本文评估了三种主流分词算法:字节对编码(BPE)、WordPiece和SentencePiece(Unigram),并比较其在杜仲喀语上的训练性能与效果。使用子词密度、持续词比例、归一化序列长度和执行时间等指标进行评估。结果表明,尽管三者均有潜力,但SentencePiece在多项指标中表现最佳,适合杜仲喀语分词,为后续该语言的大模型研究奠定基础。本研究强调了为低资源语言定制分词方案的重要性。

原文摘要 · Abstract (English)

Large Language Models (LLMs) are gaining popularity and improving rapidly. Tokenizers are crucial components of natural language processing, especially for LLMs. Tokenizers break down input text into tokens that models can easily process while ensuring the text is accurately represented, capturing its meaning and structure. Effective tokenizers enhance the capabilities of LLMs by improving a model's understanding of context and semantics, ultimately leading to better performance in various downstream tasks, such as translation, classification, sentiment analysis, and text generation. Most pre-trained tokenizers are suitable for high-resource languages like English but perform poorly for low-resource languages. Dzongkha, Bhutan's national language spoken by around seven hundred thousand people, is a low-resource language, and its linguistic complexity poses unique NLP challenges. Despite some progress, significant research in Dzongkha NLP is lacking, particularly in tokenization. This study evaluates the training and performance of three common tokenization algorithms in comparison to other popular methods. Specifically, Byte-Pair Encoding (BPE), WordPiece, and SentencePiece (Unigram) were evaluated for their suitability for Dzongkha. Performance was assessed using metrics like Subword Fertility, Proportion of Continued Words, Normalized Sequence Length, and execution time. The results show that while all three algorithms demonstrate potential, SentencePiece is the most effective for Dzongkha tokenization, paving the way for further NLP advancements. This underscores the need for tailored approaches for low-resource languages and ongoing research. In this study, we presented three tokenization algorithms for Dzongkha, paving the way for building Dzongkha Large Language Models.

分词器低资源语言杜仲喀语LLM

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。