发现最优计算效率下模型大小应随字节而非词元数扩展
Compute Optimal Tokenization
- 通过988个不同压缩率的隐式分词模型，系统研究词元粒度影响
- 计算最优时参数量与数据字节数成正比，非传统词元数比例
- 最优压缩率随算力增加而降低，适用于多种语言和分词方式
缩放定律可优化数据量与语言模型规模的匹配，但数据单元——词元——对这一关系的影响仍不明确。本文系统研究了由压缩率（即每词元平均字节数）控制的词元信息粒度对缩放趋势的影响。我们训练了988个参数范围在50M至7B之间的隐式分词模型（BLT），可自由设定所需压缩率，从而将研究范围拓展至超过4.57字节/词元（常见BPE分词器所得）。实验表明，在计算最优配置下，模型参数量与以字节计的数据量成正比，而非通常认为的词元数比例（Kaplan et al., 2020；Hoffmann et al., 2022）。此外，我们发现最优压缩率不同于BPE所得，并随算力增加而下降。该结论在隐式与子词分词、以及除英语外的语言中均成立，为语言模型开发者选择最大化算力效率的分词方案提供指导。
原文摘要 · Abstract (English)
Scaling laws enable the optimal selection of data amount and language model size, yet the impact of the data unit, the token, on this relationship remains underexplored. In this work, we systematically investigate how the information granularity of tokens, controlled by the compression rate (i.e., average bytes of text per token), affects scaling trends. We train 988 latent tokenized models (BLT) ranging from 50M to 7B parameters that enable setting the desired compression rate. This flexibility allows us to study the role of compression rate well beyond 4.57 bytes per token obtained with a popular BPE tokenizer. Our experiments reveal that in compute-optimal configurations, model parameter counts scale proportionally to data size measured in bytes, not in tokens as commonly perceived (Kaplan et al., 2020; Hoffmann et al., 2022). Furthermore, we discover that the optimal compression rate differs from the one obtained with BPE and decreases with compute. These findings generalize to both latent and subword tokenization, as well as to languages other than English, guiding language model developers on tokenization scheme selection for maximal compute efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。