为多语言大模型设计高效分词器，提升跨语言性能与推理速度。
MUTANT: A Recipe for Multilingual Tokenizer Design
- 基于语言感知预分词与子词/多词联合训练，优化分词策略。
- 在23种语言上平均熵值提升39.5%，推理吞吐量提高44%。
- 适合印度语种及多语言场景下的大模型部署使用。
分词器对大语言模型的性能、训练效率和推理成本至关重要。由于不同书写系统和丰富的形态变化，多语言分词器的设计尤为困难。尽管子词方法（如字节对编码，BPE）被广泛采用，其在多语言环境中的有效性仍待深入探索。本文提出MUTANT，一种构建多语言分词器的系统性方案，包含精心设计的词汇表与训练数据、语言感知的预分词机制，以及子词与多词联合训练策略。我们进一步推出MUTANT-Indic，专为印度多语言大模型设计的分词器，在英语、22种印度语言及代码数据上评估，相比LLaMA4平均熵值提升39.5%，相比当前最佳模型Sutra提升18%。这带来44%的推理吞吐量提升，同时保持英文和印地语基准测试性能相当。通过详尽消融实验，验证了训练数据规模、词汇量、合并技术与预分词策略的稳健性。
原文摘要 · Abstract (English)
Tokenizers play a crucial role in determining the performance, training efficiency, and the inference cost of Large Language Models (LLMs). Designing effective tokenizers for multilingual LLMs is particularly challenging due to diverse scripts and rich morphological variation. While subword methods like Byte Pair Encoding (BPE) are widely adopted, their effectiveness in multilingual settings remains underexplored. We present MUTANT, a recipe for building multilingual tokenizers, with careful vocabulary and training data design, language-aware pre-tokenization, and subword and multiword aware training. We also introduce MUTANT-Indic, a tokenizer for India-specific multilingual LLMs, that produces linguistically coherent tokens and achieves state-of-the-art performance. Evaluated across English, 22 Indian languages and code data, our tokenizer improves the average fertility score by 39.5%$ over LLaMA4 and by 18% over Sutra (the current best). This translates to 44% improvement in inference throughput over LLaMA4 while maintaining comparable performance on English and Indic benchmarks. We present detailed ablations across tokenizer training data size, vocabulary size, merging techniques, and pre-tokenization strategies, demonstrating the robustness of our design choices.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。