不同文字系统在相同模型中处理效率差异巨大，揭示了多语言模型的隐性不平等。
The Script Tax: Measuring Tokenization-Driven Efficiency and Latency Disparities in Multilingual Language Models
- 对比同语言内容的两种书写系统，发现分词碎片化程度影响模型效率
- 高碎片化书写系统导致推理速度下降16.5倍，信息成本上升近50%
- 适合关注多语言模型公平性与分词设计的研究者
预训练多语言模型常被视为与文字系统无关，但其分词器对某些书写系统存在系统性代价。我们通过比较两种语义相同但拼写形式不同的变体，量化这种‘文字税’。在mBERT和XLM-R模型中，高碎片化书写系统使每词分词数从2.10-2.35增至6.73-6.85（约3.4倍），推理速度从每秒3.8句降至0.23句（慢16.5倍）。使用每字符比特数（BPC）避免子词碎片化带来的“负对数似然悖论”，发现信息成本显著上升：mBERT从8.06升至9.65（+19.7%），XLM-R从12.19升至17.94（+47.1%）。往返转换校验（CER_rt=0.31）表明这些差距反映的是文字系统依赖的处理差异，而非映射噪声。结果凸显分词是多语言NLP中不平等的关键来源，推动更注重文字系统的分词与预训练设计。
原文摘要 · Abstract (English)
Pretrained multilingual language models are often assumed to be script-agnostic, yet their tokenizers can impose systematic costs on certain writing systems. We quantify this script tax by comparing two orthographic variants with identical linguistic content. Across mBERT and XLM-R, the higher-fragmentation orthography shows a ~3.4x increase in fertility (6.73-6.85 vs. 2.10-2.35 tokens/word), leading to a 16.5x inference slowdown (0.23 vs. 3.8 sentences/second) on identical hardware. Using bits per character (BPC) to avoid the "NLL paradox" from subword fragmentation, we find a substantial increase in information cost: +19.7% for mBERT (8.06->9.65) and +47.1% for XLM-R (12.19->17.94). A round-trip conversion check (CER_rt=0.31) suggests these gaps reflect orthography-conditioned processing rather than mapping noise. Our results highlight tokenization as a key source of inequity in multilingual NLP and motivate script-aware tokenization and pretraining.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。