通过字节级模拟，拆解了子词分词对大模型训练的贡献。
Decoupling the Benefits of Subword Tokenization for Language Model Training via Byte-level Simulation
- 在字节级预训练中分离子词分词的影响，控制变量分析。
- 子词边界作为先验知识可提升训练吞吐量和模型性能。
- 适合关注预训练效率与分词设计的研究者阅读。
子词分词是现代大语言模型的关键环节，但其对训练效率和模型性能的具体贡献仍不明确。本文通过构建受控的字节级预训练流程，将子词分词的影响进行解耦，并在多个维度（如样本吞吐量、词表规模、子词边界语言先验）上验证假设。通过在字节级设置中模拟这些效应，我们更清晰地揭示了为何子词模型优于原始字节模型，并为未来字节级与子词模型的预训练优化提供了洞见。实验表明，更高的训练吞吐量以及子词边界作为显式先验或归纳偏置的整合，是关键因素。
原文摘要 · Abstract (English)
Subword tokenization is an essential part of modern large language models (LLMs), yet its specific contributions to training efficiency and model performance remain poorly understood. In this work, we decouple the effects of subword tokenization by isolating them within a controlled byte-level pretraining pipeline. We formulate and test hypotheses across various dimensions, including sample throughput, vocabulary scaling, and the linguistic prior of subword boundaries. By simulating these effects in a byte-level setting, we refine our understanding of why subword models outperform raw byte models and offer insights to improve the pretraining of future byte-level and subword models. Specifically, our experiments highlight the critical role of increased training throughput and the integration of subword boundaries as either explicit priors or inductive biases.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。