非洲语言在大模型中被过度分词，导致成本和延迟飙升。
The African Language Tax: Quantifying the Cost, Latency, and Context Penalty of Tokenizing African Languages in Frontier LLMs
- 通过对比20种非洲语言的分词效率，发现其平均分词量是英文的1.88倍。
- 部分语言如尼科文和埃塞俄比亚文分词量达英文的8.92倍，成本翻8倍。
- 该问题影响非洲开发者，工具已开源以推动公平性改善。
主流大模型按令牌计费、计算延迟并消耗上下文容量。然而，分词器对不同语言分配的子词令牌数不均，导致某些语言使用者在模型调用前就面临结构性代价。尽管多语言场景中的此现象已被记录，但非洲语言在企业级部署经济与认知上下文容量层面的系统性测量仍缺失。本研究基于平行语料库，隔离内容影响，在涵盖五种语系、三种文字（拉丁、吉兹/埃塞俄比亚文、尼科文）的20种非洲语言上进行评估，使用FLORES-200+主语料库（19种）、仅以MAFAND-MT衡量的尼日利亚皮钦语。覆盖11种前沿及开源分词器（FLORES-200+），所有非洲语言的分词量均高于英文（中位数为GPT-5 / o200k_base的1.88倍，尼科文达8.92倍）；埃塞俄比亚文与尼科文文字的惩罚最严重（7–9倍），且跨语料库结果高度一致（FLORES vs SIB-200 Pearson r = 0.9998）。换算为部署成本，最高达英文8.9倍推理开销，生成延迟亦乘以相同倍数（尼科文对英文在GPT-5上，阿姆哈拉语达7.4倍），有效上下文窗口最低仅英文的11%。目前表现最优的非洲语言分词器Gemma 4将平均分词溢价从cl100k_base的3.31倍降至2.38倍，但无法彻底消除该惩罚。我们发布了开源测量工具afri-fertility、公开排行榜、结果数据集及缓解指南，供非洲开发者使用。这一代价主要落在最无力承担的语言群体身上，构成直接编码于子词词汇表的数字鸿沟。
原文摘要 · Abstract (English)
Commercial large language models bill, scale latency, and budget context per token. Yet tokenizers assign more subword tokens to the same meaning in some languages than in others, so speakers of languages with high token-fertility pay a structural penalty before a model is ever invoked. This penalty is documented for multilingual settings in general, but it has not been measured systematically for African languages at the level of enterprise deployment economics and cognitive context capacity. We measure it across 20 African languages spanning five language families and three scripts (Latin, Ge'ez/Ethiopic, N'Ko; 19 appear in the primary FLORES-200+ corpus, with Nigerian Pidgin measured via MAFAND-MT only), using parallel corpora so that the language effect is isolated from content. Across 11 frontier and open tokenizers on FLORES-200+, every African language carries a tokenization premium above English (median 1.88x on GPT-5 / o200k_base, up to 8.92x for N'Ko); the penalty is largest for Ethiopic and N'Ko scripts (reaching 7-9x) and is near-invariant across corpora (FLORES vs SIB-200 Pearson r = 0.9998). Translated into deployment terms, this results in up to 8.9x inference cost and an equivalent generation-latency multiplier (N'Ko vs English on GPT-5; 7.4x for Amharic), and as little as 11% of English's effective context window. The best currently available tokenizer for African languages, Gemma 4, reduces the mean premium from 3.31x (cl100k_base) to 2.38x, but no tokenizer eliminates the penalty. We release an open measurement tool (afri-fertility), a public leaderboard, a results dataset, and mitigation guidance for African builders. The penalty falls hardest on the languages whose speakers can least afford it, a digital divide encoded directly into the subword vocabulary.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。