研究发现梵语每令牌表达信息量是英语的2倍，更省计算资源。
Is Sanskrit the most token-efficient language? A quantitative study using GPT, Gemini, and SentencePiece
- 用梵语、英语、印地语诗文对比不同分词器效率
- 梵语在统一分词器下比英语少约一半令牌数
- 主流模型仍难完全捕捉梵语紧凑性，可能影响非英语用户
令牌是大语言模型的基本单位。模型依赖分词器将文本切分为令牌，而分词方式直接决定计算与推理成本。梵语作为古老语言，因其形态和语法规则，被认为能以更少令牌传递更多信息，但此前缺乏量化验证。本研究使用包含701对《薄伽梵歌》诗句的平行数据集（含梵语、英语、印地语及梵语英译），测试了SentencePiece、GPT系列旧版分词器，以及最新Gemini和GPT分词器。采用令牌数、每令牌字符数（令牌效率）、每字符令牌数（令牌成本）等指标。结果表明，在无偏的SentencePiece基准下，梵语令牌数比英语/印地语低约2倍；英语/印地语对梵语注释的翻译使令牌数增加约20倍。相较GPT cl100k base（旧版），GPT o200k base（GPT-4o所用）和Gemini（最新版）显著降低偏差，但仍未能完全捕捉梵语的紧凑特性。这表明非英语用户可能存在隐性成本惩罚，导致开销上升。该研究为未来分词器设计提供依据，揭示梵语在高效编码中的潜力，可降低训练与推理成本。代码与数据集已开源：https://github.com/anshulkr713/sanskrit-token-efficiency。
原文摘要 · Abstract (English)
Tokens are the basic units of Large Language Models (LLMs). LLMs rely on tokenizers to segment text into these tokens, and tokenization is the primary determinant of computational and inference cost. Sanskrit, one of the oldest languages, is hypothesized to express more meaning per token due to its morphology and grammar rules; however, no prior work has quantified this. We use a dataset of 701 parallel verses of the Bhagavad Gita, which comprises three languages-Sanskrit, English, and Hindi along with transliteration of Sanskrit into English. We test tokenizers including SentencePiece (SPM), older GPT models, and the latest generation tokenizers from Gemini and GPT. We use metrics of token count, characters per token (token efficiency), and tokens per character (token cost). Results show a ~2x difference in token counts between Sanskrit and English/Hindi under the unbiased SPM baseline. English/Hindi translations of Sanskrit commentary resulted in an approximately 20x increase in token count. GPT o200k base (latest, used by GPT-4o) and Gemini (latest) reduce bias by a significant degree compared to GPT cl100k base (used until GPT-4), but still fail to fully capture Sanskrit's compactness. This matters because there might be a penalty bias for non-English users, which inflates the token count. This research provides a foundation for improving future tokenizer design and shows the potential of Sanskrit for highly compact encoding, saving on cost while speeding up training and inference. The code and dataset are available at https://github.com/anshulkr713/sanskrit-token-efficiency
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。