arXiv:2608.21384cs.CLcs.AI2026-08

乌克兰语在主流分词器中消耗更多字节,导致成本与上下文容量劣势。

Beyond Two Bytes per Letter: Tokenization Overhead in Cyrillic AI Systems

  • 对比九种分词器,乌克兰语字节开销高达英语的1.68至2.21倍
  • 新分词器设计可将乌克兰/英语比例从2.22降至1.30,减少近40%差异
  • 适合关注多语言公平性、低资源语言优化的研究者与工程师

现代多语言分词器对乌克兰语等非主流西里尔字母语言的切分比英文更细,造成成本与上下文容量差距。我们在九种生产级分词器和五种具有标准西里尔与拉丁表示的语言上量化了该开销,涵盖837万词形。在基准语料库上,乌克兰语在现代分词器下开销达68%-121%,在旧版cl100k下达220%,以全文本肥沃度(full-text fertility)衡量。开销与西里尔词汇分配呈负相关(斯皮尔曼秩相关系数rho = -0.536,p = 0.215,n = 7),但不显著。我们评估两种缓解策略:LLMLingua-2在1,536个商品、145个查询的电商RAG基准上将输入长度减少47%-49%,80个可检索案例无压缩损失;一个基于字节的平衡BPE分词器(20万词表上限,实际收敛于158,184项),将保留集中的乌/英比例从2.22倍降至1.30倍。罗马化使乌克兰语在多数分词器中词元数增加2%-19%。跨五种语言,分词效率偏向网络数据更丰富的脚本。结果表明训练数据分配影响西里尔字母分词开销,且可在推理与分词器设计阶段进行缓解。

原文摘要 · Abstract (English)

Modern multilingual tokenizers often fragment Ukrainian and other underrepresented Cyrillic-script languages more heavily than English, creating disparities in cost and context capacity. We quantify this overhead across nine production tokenizers and five languages with standardized Cyrillic and Latin representations, covering 8.37 million word forms. On a corpus benchmark, Ukrainian shows 68-121% token overhead on modern tokenizers and 220% on the older cl100k, measured through full-text fertility on the BrUK and Brown corpora. Overhead is negatively associated with Cyrillic vocabulary allocation in the subset with independently verified English baselines, although the association is not statistically significant (Spearman rho = -0.536, p = 0.215, n = 7). We evaluate two mitigation strategies. LLMLingua-2 reduces Ukrainian input length by 47-49% on an e-commerce RAG benchmark of 1,536 products and 145 queries, with no compression-induced value losses among 80 retrievable cases. A balanced byte-level BPE tokenizer trained with a 200K vocabulary cap, converging at 158,184 actual entries, reduces the held-out UK/EN ratio from 2.22x to 1.30x. Romanization increases Ukrainian token counts by 2-19% on most tokenizers. Across the five languages, tokenization efficiency favors the script more prevalent in web data. These findings indicate that training data allocation contributes to Cyrillic tokenization overhead and that mitigation is possible at both inference and tokenizer-design stages.

分词器多语言西里尔字母效率优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。