为土耳其语设计了一种结合词法与音系规则的混合分词方法,显著提升分词准确率。
Tokens with Meaning: A Hybrid Tokenization Approach for Turkish
- 融合词典驱动分词、音系归一化和子词回退机制
- 在TR-MMLU上达到90.29%的词素对齐率和85.80%纯净对齐率
- 适合需要精准词法结构理解的土耳其语下游任务
分词方式影响语言模型对形态与意义的理解。现有基于频率的子词分词器(如Byte Pair Encoding和WordPiece)常将形态丰富且黏着性强的土耳其语过度切分,模糊词素边界。本文提出一种基于语言学知识的混合分词方法:(i) 基于词典的词根与词缀分割;(ii) 音系归一化,将所有形变映射到统一标识符;(iii) 受控子词回退以覆盖未登录词。所发布词汇表包含22,231个词根标记对应20,000个规范词根标识符(前导空格标记词边界),72个词缀标识符覆盖177个表面形式,以及12,696个子词单元;正写法标记保留大小写信息而不膨胀词表。在TR-MMLU数据集上,使用两种语言对齐指标评估:土耳其词占比(TR~ %)为90.29%,纯词占比(Pure~ %)为85.80%,显著优于多个通用分词器。进一步通过下游句向量任务验证实用性,在严格随机初始化控制下,TurkishTokenizer在土耳其语STS基准上超越所有基线,于MTEB-TR中取得最强平均表现,并在基于质心代理的TurBLiMP上实现最高平均准确率。
原文摘要 · Abstract (English)
Tokenization shapes how language models perceive morphology and meaning in NLP, yet widely used frequency-driven subword tokenizers (e.g., Byte Pair Encoding and WordPiece) can fragment morphologically rich and agglutinative languages in ways that obscure morpheme boundaries. We introduce a linguistically informed hybrid tokenizer for Turkish that combines (i) dictionary-driven morphological segmentation (roots and affixes), (ii) phonological normalization that maps allomorphic variants to shared identifiers, and (iii) a controlled subword fallback for out-of-vocabulary coverage. Concretely, our released Turkish vocabulary contains 22,231 root tokens mapped to 20,000 canonical root identifiers (with leading spaces to mark word boundaries), 72 affix identifiers that cover 177 allomorphic surface forms, and 12,696 subword units; an orthographic case token preserves capitalization without inflating the vocabulary. We evaluate tokenization quality on the TR-MMLU dataset using two linguistic alignment metrics: Turkish Token Percentage (TR~\%), the proportion of produced tokens that correspond to Turkish lexical/morphemic units under our lexical resources, and Pure Token Percentage (Pure~\%), the proportion of tokens aligning with unambiguous root/affix boundaries. The proposed tokenizer reaches 90.29\% TR~\% and 85.80\% Pure~\% on TR-MMLU, substantially exceeding several general-purpose tokenizers. We further validate practical utility with downstream sentence embedding benchmarks under a strict \emph{random initialization} control to isolate tokenizer inductive bias. Across four matched models (TurkishTokenizer, CosmosGPT2, Mursit, and Tabi), TurkishTokenizer outperforms all baselines on the Turkish STS Benchmark and achieves the strongest overall average on MTEB-TR. It also yields the strongest average accuracy on the TurBLiMP under a centroid-based proxy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。