扩大输入词表能显著提升模型性能，且无需额外成本。
Over-Tokenized Transformer: Vocabulary is Generally Worth Scaling
- 分离输入输出词表，用多字节词元扩大输入词表
- 输入词表越大，训练损失越低，性能持续提升
- 适合关注词元化设计与模型效率的开发者
词元化是大语言模型的基础组件，但其对模型扩展和性能的影响尚未充分探索。本文提出过词元化Transformer框架，通过解耦输入与输出词表来提升语言建模性能。具体地，该方法扩大输入词表以利用多字节词元。大量实验揭示输入词表大小与训练损失之间存在对数线性关系，表明更大的输入词表在不同模型规模下均能持续提升性能。使用大输入词表，可在无额外成本情况下达到双倍模型规模基线的性能。研究强调了词元化在缩放定律中的重要性，并为分词器设计提供了实用洞见，推动更高效、强大的大语言模型发展。
原文摘要 · Abstract (English)
Tokenization is a fundamental component of large language models (LLMs), yet its influence on model scaling and performance is not fully explored. In this paper, we introduce Over-Tokenized Transformers, a novel framework that decouples input and output vocabularies to improve language modeling performance. Specifically, our approach scales up input vocabularies to leverage multi-gram tokens. Through extensive experiments, we uncover a log-linear relationship between input vocabulary size and training loss, demonstrating that larger input vocabularies consistently enhance model performance, regardless of model size. Using a large input vocabulary, we achieve performance comparable to double-sized baselines with no additional cost. Our findings highlight the importance of tokenization in scaling laws and provide practical insight for tokenizer design, paving the way for more efficient and powerful LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。