131K词表的印地语友好分词器，兼顾多语言压缩效率。
BrahmicTokenizer-131K: An Indic-Capable Drop-In Replacement for o200k_base
- 通过裁剪与精准重构，将o200k_base词表适配为支持婆罗米系文字的131,072词表。
- 在印地语文本上比现有工具少26.7%的分词数，奥迪亚语压缩比达4.31倍。
- 兼容原接口，非印地语性能不降反升，适合多语言模型部署。
我们提出BrahmicTokenizer-131K，一个131,072词表的字节级BPE分词器，在保持OpenAI o200k_base对英语、欧洲语言和代码良好压缩性能的同时，填补了婆罗米系文字的压缩差距。通过两阶段改造：(1) 剪除9个无关书写系统，将200,019个词元缩减至131,072；(2) 利用线性规划在9个婆罗米类Unicode区块中手术式填充2,372个空闲词元槽位。预分词器、解码器及合并规则均沿用o200k_base，实现接口级无缝替换。在2700万篇公开印地语预训练文本（28.4亿词，46.21 GB）上，其分词量比Mistral-Nemo Tekken / Sarvam-m减少26.7%，各语言节省率15.79%（泰米尔语）至76.79%（奥迪亚语，压缩比4.31倍）。奥迪亚语优势源于原模型无奥里亚块词元，本工作新增725个。非印地语内容上，该分词器在英语词密度（1.235 vs 1.232）和HumanEval、MBPP、GSM8K任务上优于Tekken/Sarvam-m（提升4.0%-14.2%）。在14个分词器基准测试中，它是唯一在131K词表下同时具备婆罗米、英语、欧洲语言、代码和数学竞争力的方案。其他专用分词器（Sarvam-30B、Sarvam-1、MUTANT-Indic）虽在印地语上更优，但牺牲英语性能（差15.9%）和代码/数学压缩（差26%-33%）。代码与模型已开源于Apache 2.0许可。
原文摘要 · Abstract (English)
We present BrahmicTokenizer-131K, a 131,072-vocabulary byte-level BPE tokenizer that closes the Brahmic compression gap at the 131K-vocabulary class while preserving the English, EU-language, and code compression of OpenAI's o200k_base. We construct it through a two-stage retrofit: (1) a script-prune crop that reduces 200,019 tokens to 131,072 by removing nine out-of-scope writing systems, and (2) a surgical retrofit of 2,372 corpus-dead vocabulary slots determined by linear-programming allocation across nine Brahmic Unicode blocks. The pre-tokenizer, decoder, and inherited merge rules are unchanged from o200k_base, making BrahmicTokenizer-131K a drop-in replacement at the tokenizer interface. On 27 million documents of public Indic pretraining text (2.84 billion words, 46.21 GB), BrahmicTokenizer-131K produces 26.7% fewer tokens than Mistral-Nemo Tekken / Sarvam-m at the same vocabulary budget, with per-language savings of 15.79% (Tamil) to 76.79% (Odia, a 4.31x compression ratio). The Odia advantage is mechanistically explained by Tekken/Sarvam-m containing zero Oriya-block tokens; our surgery added 725. On non-Indic content, BrahmicTokenizer-131K matches o200k_base's English fertility (1.235 vs 1.232 tokens/word) and beats Tekken/Sarvam-m by 4.0-14.2% on HumanEval, MBPP, and GSM8K. Across our 14-tokenizer benchmark, it is the only tokenizer simultaneously competitive on Brahmic, English, EU, code, and math at the 131K budget. Specialist tokenizers at other vocab classes (Sarvam-30B, Sarvam-1, MUTANT-Indic) achieve better Indic compression at the cost of non-Indic performance: Sarvam-1's English fertility is 15.9% worse and its code/math compression 26-33% worse than ours. We release the artifact under Apache 2.0 at https://huggingface.co/theschoolofai/BrahmicTokenizer-131K.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。