拼音符号被误作字母，导致多语言分词器效率下降
Vowel Signs Are Not Letters: A Pre-tokenization Ceiling on Multilingual Tokenizer Fertility
- 用字节级BPE分词时，元音符号被当作独立字符拆分，限制分词效果
- 26种语言中17个辅音音节文字受此影响，分词量最多翻9倍
- 修复后模型在尼泊尔语上性能提升，适合多语言NLP研究者
使用HuggingFace字节级预分词器的字节级BPE分词器继承了GPT-2的单词正则表达式，将单词定义为一个或多个Unicode字母（\p{L}+）。在辅音音节文字中，元音以组合标记形式书写，因此该模式会在每个元音符号处拆分单词。由于BPE仅在预分词内合并，这些拆分会持续存在于训练过程中，无论词汇量或语料组成如何。我们将其形式化为一种无需训练的分词率下界。在来自平行语料库的26种语言中，17个辅音音节文字均受影响，分词率从1.47倍（藏语）到9.02倍（泰语）不等；而拉丁、西里尔、韩文和汉字文字则保持1.00倍。5种语言中，仅预分词方式不同的分词器对比例与预测下界相差不超过2.2%，尼泊尔语的分词量分别为4.78和1.58。当尼泊尔语在训练语料中占比从5%升至95%时，故障分词器仅变化1.7%，而修复版本变化33.9%，表明结构性瓶颈与数据不足可区分。训练三个仅分词器不同的268M模型，修复版在同等算力下尼泊尔语的每字节比特数降低4.43%，即使提供1.59倍计算资源仍领先。对3,479个HuggingFace仓库的普查显示，仅包含字母的单词类别出现在63.3%的热门文本生成模型中，占其下载量的72.5%。GPT-4o的o200k模式已采用标记感知的单词类别，修复方法本身已有先例。我们量化其价值，展示如何仅通过症状识别缺失，确定适用脚本范围，测量部署广度，并发布一个含65,536条目的尼泊尔语-英语分词器及工具包，可在笔记本电脑上从公开数据复现所有数值。
原文摘要 · Abstract (English)
Byte-level BPE tokenizers that use the HuggingFace ByteLevel pre-tokenizer inherit GPT-2's word regex, where a word is defined as \p{L}+, one or more Unicode letters. In abugida scripts, vowels are written as combining marks; this pattern therefore splits each word at every vowel sign. Since BPE merges only within a pre-token, those splits persist through training regardless of vocabulary size or corpus composition. We formalise this effect as a training-free lower bound on fertility. Across 26 languages from a parallel corpus, every one of the 17 abugidas is affected, ranging from 1.47x (Tibetan) to 9.02x (Thai), whereas Latin, Cyrillic, Hangul, and Han show exactly 1.00x. For 5 languages, matched tokenizer pairs that differ only in this character class fall within 2.2% of the predicted floor, scoring 4.78 versus 1.58 tokens per word on Nepali. When the Nepali share of the training corpus is swept from 5% to 95%, the broken tokenizer barely shifts at all (1.7%) while the fixed one shifts 33.9%, which separates a structural ceiling from a data shortage without needing to inspect any code. We train three 268M models that differ only in their tokenizer; the fixed variant achieves 4.43% lower held-out Nepali bits per byte at equal compute, and it still leads when given the same bytes with 1.59x the compute. A census of 3,479 HuggingFace repositories finds the letters-only word class present in 63.3% of the most-downloaded text-generation models, accounting for 72.5% of their downloads. GPT-4o's o200k pattern already uses a mark-aware word class, making the repair itself prior art. We quantify its value, show how to recognise its absence from symptoms alone, map which scripts it reaches, measure how widely it is deployed, and release a 65,536-entry Nepali-English tokenizer with a harness that regenerates every number here from public data on a laptop.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。