低收入国家语言用户用大模型成本高6倍，且效果差，形成双重劣势。
Double Jeopardy and Climate Impact in the Use of Large Language Models: Socio-economic Disparities and Reduced Utility for Non-English Speakers
- 按令牌计费的模型对非英语语言更不友好，导致成本高出4-6倍。
- 低资源语言翻译任务表现差，用户体验与英语使用者差距显著。
- 算法设计需考虑公平性，避免加剧语言与气候的双重不平等。
人工智能，特别是大型语言模型（LLMs），本可缩小语言与信息鸿沟，惠及发展中国家经济。然而，基于FLORES-200、FLORES+、Ethnologue和世界发展指标数据的分析显示，这些收益主要惠及英语使用者。使用OpenAI GPT API时，来自低收入和下中等收入国家的语言用户因输入处理方式（分词）导致成本更高——约15亿人面临4至6倍于英语用户的费用。模型在低资源语言上的性能差异显著，而按令牌计费机制进一步放大了获取成本与使用效用的不平等。通过翻译任务质量作为代理指标，我们发现低资源语言表现较差，形成‘双重劣势’：成本高且效果差。此外，低资源语言分词碎片化还直接影响碳排放。这凸显了必须推动更公平的算法开发，以惠及所有语言群体。
原文摘要 · Abstract (English)
Artificial Intelligence (AI), particularly large language models (LLMs), holds the potential to bridge language and information gaps, which can benefit the economies of developing nations. However, our analysis of FLORES-200, FLORES+, Ethnologue, and World Development Indicators data reveals that these benefits largely favor English speakers. Speakers of languages in low-income and lower-middle-income countries face higher costs when using OpenAI's GPT models via APIs because of how the system processes the input -- tokenization. Around 1.5 billion people, speaking languages primarily from lower-middle-income countries, could incur costs that are 4 to 6 times higher than those faced by English speakers. Disparities in LLM performance are significant, and tokenization in models priced per token amplifies inequalities in access, cost, and utility. Moreover, using the quality of translation tasks as a proxy measure, we show that LLMs perform poorly in low-resource languages, presenting a ``double jeopardy" of higher costs and poor performance for these users. We also discuss the direct impact of fragmentation in tokenizing low-resource languages on climate. This underscores the need for fairer algorithm development to benefit all linguistic groups.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。