动态调整词汇表提升大模型预训练效率
Scaling LLM Pre-training with Vocabulary Curriculum
- 通过熵引导的词汇扩展与模型优化交替进行
- 词汇量增大时，训练效率呈对数线性提升
- 适合关注高效预训练与动态分词的研究者
现代语言模型使用预设的静态词汇表，与人类语言学习中词汇随认知发展逐步积累的现象不一致。为此，我们提出词汇课程学习（vocabulary curriculum learning），通过在熵引导的词汇扩展与模型优化之间交替进行，实现了相对于词汇量的对数线性缩放增益。该方法使模型能够在不同粒度的分词下学习可迁移的表示。自然地，该过程形成最优计算分配模式：长片段捕获可预测内容，短片段聚焦复杂、难预测的上下文。小规模GPT模型实验验证了训练效率的提升，证实了动态分词的有效性。代码已开源，未来将拓展至更大模型和多样化领域。
原文摘要 · Abstract (English)
Modern language models rely on static vocabularies, fixed before pretraining, in contrast to the adaptive vocabulary acquisition observed in human language learning. To bridge this gap, we introduce vocabulary curriculum learning, an approach that improves pretraining efficiency with log-linear scaling gains relative to vocabulary size. Our method alternates between entropy-guided vocabulary expansion and model optimization, enabling models to learn transferable representations across diverse tokenization granularities. This approach naturally gives rise to an optimal computation allocation pattern: longer tokens capture predictable content, while shorter tokens focus on more complex, harder-to-predict contexts. Experiments on small-scale GPT models demonstrate improved scaling efficiency, reinforcing the effectiveness of dynamic tokenization. We release our code to support further research and plan to extend our experiments to larger models and diverse domains.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。