训练数据超过150GB后分词质量提升有限，需警惕数据冗余。
How Much is Enough? The Diminishing Returns of Tokenization Training Data
- 对比不同数据量（1GB~900GB）训练分词器，发现收益递减。
- 英语数据超150GB后分词质量提升趋缓，俄语约200GB开始饱和。
- 揭示预分词阶段是限制性能的关键瓶颈，适合优化训练流程者阅读。
分词是自然语言处理的关键初始步骤，受多种参数影响，包括分词算法、词汇表大小、预分词策略、推理策略和训练数据集。本文研究了一个常被忽视的超参数——分词器训练数据规模。我们在英语数据（1GB至900GB）上训练了BPE、UnigramLM和WordPiece分词器，覆盖多种词汇表大小。结果表明，当训练数据超过约150GB后，分词质量提升出现明显递减，提示通过增加数据改善分词性能存在实际上限。我们分析该现象，归因于预分词阶段的约束。进一步在语言类型迥异的俄语文本上验证，发现俄语在200GB后出现收益递减，较英语高出约33%。这些发现为优化分词流程提供重要依据，可减少大规模数据训练的算力开销，并指明未来分词算法改进方向。
原文摘要 · Abstract (English)
Tokenization, a crucial initial step in natural language processing, is governed by several key parameters, such as the tokenization algorithm, vocabulary size, pre-tokenization strategy, inference strategy, and training data corpus. This paper investigates the impact of an often-overlooked hyperparameter, tokenizer training data size. We train BPE, UnigramLM, and WordPiece tokenizers across various vocabulary sizes using English training data ranging from 1GB to 900GB. Our findings reveal diminishing returns as training data size increases beyond roughly 150GB, suggesting a practical limit to the improvements in tokenization quality achievable through additional data. We analyze this phenomenon and attribute the saturation effect to constraints introduced by the pre-tokenization stage. We then demonstrate the extent to which these findings can generalize by experimenting on data in Russian, a language typologically distant from English. For Russian text, we observe diminishing returns after training a tokenizer from 200GB of data, which is approximately 33% more than when training on English. These results provide valuable insights for optimizing the tokenization process by reducing the compute required for training on large corpora and suggest promising directions for future research in tokenization algorithms.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。