arXiv:2608.09046cs.CLcs.CY2026-08中稿 · IJCAI

发现低资源语言文本在大模型中消耗更多令牌,导致成本与性能下降。

Measuring the Tokenization Premium: A Cost Audit for Underserved Language Communities

论文配图:Measuring the Tokenization Premium: A Cost Audit for Underserved Language Communities
图 1 · 摘自论文原文
  • 构建可复现的基准测试,评估多种分词器对5种语言的技术内容分词差异。
  • 孟加拉语在GPT-4o下需1.56倍于英语的令牌数,有效上下文窗口缩小至82k。
  • 即使使用拉丁字母的约鲁巴语也存在2.37倍分词溢价,说明不平等非仅由文字系统造成。

大型语言模型日益作为通用教育与技术辅助系统部署,但其底层基础设施对语言并不公平。一个未充分探讨的差异来源是分词:语义等价内容在不同语言间可能需要显著不同的令牌数量,影响API成本、延迟及调用前可用上下文长度。我们提出分词公平性审计(TEA),一个可复现的基准测试,用于衡量技术辅导内容中的分词溢价。TEA评估三种常用分词器——GPT-4o的o200k base、Qwen2.5-7B和Mistral-7B——在120项从英语翻译成孟加拉语、印地语、阿拉伯语、泰米尔语和约鲁巴语的Python调试语料上的表现。孟加拉语和印地语为验证重点,其余语言用于跨书写系统与语系比较。在此语料中,孟加拉语在GPT-4o下需1.56倍于英语的令牌数,使原本128k令牌的上下文窗口有效容量降至82k英文等效量。使用Qwen2.5与Mistral分词器时,孟加拉语最多需4.5倍于英语的令牌数。约鲁巴语虽使用拉丁字母,却在GPT-4o下达到2.37倍的最高分词溢价,表明分词不公不能仅由书写系统解释。结果表明,分词可能制造可量化的经济与功能障碍,凸显应将分词视为低资源语言社区的关键公平性基础设施层,尤其在依赖低成本或离线运行的教育系统中。

原文摘要 · Abstract (English)

Large language models are increasingly deployed as general-purpose educational and technical assistance systems, but their underlying infrastructure does not treat languages equally. One underexamined source of disparity is tokenization: semantically equivalent content can require substantially different token counts across languages, affecting API cost, latency, and usable context length before a model is invoked. We introduce the Tokenization Equity Audit (TEA), a reproducible benchmark for measuring tokenization premiums in technical tutoring content. TEA evaluates three widely used tokenizers, GPT-4o's o200k base, Qwen2.5-7B, and Mistral-7B, on a 120-item Python debugging corpus translated from English into Bengali, Hindi, Arabic, Tamil, and Yoruba. Bengali and Hindi serve as the primary validated cases, while the remaining languages provide exploratory cross-script and cross-family comparisons. Across this corpus, Bengali requires (1.56\times) as many GPT-4o tokens as English, reducing a nominal 128k-token context window to an effective 82k-token English-equivalent capacity for the same semantic content. With the Qwen2.5 and Mistral tokenizers, Bengali requires up to (4.5\times) the English token count. Yoruba, despite using the Latin script, exhibits the highest GPT-4o tokenization premium at (2.37\times), indicating that tokenization inequity cannot be explained by script family alone. These results demonstrate that tokenization can create measurable economic and functional barriers, highlighting the need to treat tokenization as an equity-relevant infrastructure layer for underserved language communities, particularly where educational systems depend on low-cost or offline-capable AI tools.

分词语言公平成本优化低资源语言

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。