arXiv:2609.06690cs.CL2026-09

为泰米尔语设计的字形感知分词器,提升分词效率与模型兼容性。

A Grapheme-Aware Indic Tokenizer for Tamil: Large-Scale Training and Intrinsic Evaluation

  • 基于字形簇的逆向映射策略,避免字符拆分。
  • 在碎片化和序列效率上优于5种主流分词器。
  • 适合研究南亚语言分词或构建泰米尔语大模型的人使用。

分词是现代自然语言处理系统的基础,将原始文本转化为神经语言模型可处理的离散单元。该过程的效率直接影响词汇利用率、序列长度、计算成本及下游模型性能。尽管如BPE、WordPiece和SentencePiece等多语言分词器在多种语言中表现良好,但对构词丰富的印地语系语言(如泰米尔语)常处理不当。泰米尔语因其基于字形的书写系统,一个可见字符可能由多个Unicode码点表示,导致分词困难。本文提出一种面向泰米尔语的字形感知分词器,通过在WordPiece词汇学习前采用可逆的Unicode映射策略,保留完整的字形簇。该方法在字形层面操作而非单个码点,生成符合语言学意义的分词边界,同时保持与Transformer模型完全兼容。分词器在大规模泰米尔语语料上训练,并通过压缩效率、分词碎片化、信息密度和词汇利用率等内在评估框架进行评测。实验对比了GPT-2、mBERT、mT5、mBART和NLLB共五种常用多语言分词器。结果表明,所提分词器在碎片化与序列效率相关指标上表现最优,且达到最高的压缩比。证明了字形感知预处理在泰米尔语分词中的有效性。

原文摘要 · Abstract (English)

Tokenization forms the foundation of modern Natural Language Processing (NLP) systems by transforming raw text into discrete units that neural language models can process. The effectiveness of this process directly influences vocabulary efficiency, sequence length, computational cost, and downstream model performance. Although multilingual tokenizers such as Byte Pair Encoding (BPE), WordPiece, and SentencePiece have performed well across numerous languages, they often segment morphologically rich Indic languages inefficiently. Tamil, in particular, poses unique challenges because its grapheme-based writing system can represent a single visible character with multiple Unicode code points. In this work, we present a grapheme-aware Indic tokenizer for Tamil that preserves complete grapheme clusters through a reversible Unicode mapping strategy prior to WordPiece vocabulary learning. By operating on grapheme-level representations instead of individual Unicode code points, the tokenizer produces linguistically meaningful token boundaries while remaining fully compatible with transformer-based language models. The tokenizer is trained on a large-scale Tamil corpus and evaluated using a comprehensive intrinsic evaluation framework that measures compression efficiency, token fragmentation, information density, and vocabulary utilization. Experimental evaluation compares the proposed tokenizer against five widely used multilingual tokenizers: GPT-2, mBERT, mT5, mBART, and NLLB. The proposed tokenizer achieves the strongest performance among the evaluated tokenizers on fragmentation- and sequence-efficiency-oriented intrinsic metrics, while matching the highest observed compression ratio. These results demonstrate the effectiveness of grapheme-aware preprocessing for Tamil tokenization.

分词器泰米尔语字形语言模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。