改进大模型分词方式，让长词更优先，提升学习均衡性
LBPE: Long-token-first Tokenization to Improve Large Language Models
- 按词长逆序优先分词，长词获得更高编码优先级
- 长词频率差异缩小，训练时学习更均衡，性能普遍提升
- 适合追求分词优化的模型开发者，尤其关注长词表征
主流大语言模型采用字节对编码（BPE）处理子词单元，有效避免了词汇外问题。然而，长词虽语义丰富，但在分词后出现频率远低于短词，导致不同词粒度间学习不均衡。为此，本文提出LBPE，其在编码过程中优先处理长词，依据词长的逆排名而非词汇排名生成分词，使长词在编码中享有更高优先级。该方法显著降低了长短词间的频率差异，缓解了学习不平衡问题。在多种语言建模任务上的大量实验表明，LBPE始终优于原始BPE，充分验证了其有效性。
原文摘要 · Abstract (English)
The prevalent use of Byte Pair Encoding (BPE) in Large Language Models (LLMs) facilitates robust handling of subword units and avoids issues of out-of-vocabulary words. Despite its success, a critical challenge persists: long tokens, rich in semantic information, have fewer occurrences in tokenized datasets compared to short tokens, which can result in imbalanced learning issue across different tokens. To address that, we propose LBPE, which prioritizes long tokens during the encoding process. LBPE generates tokens according to their reverse ranks of token length rather than their ranks in the vocabulary, granting longer tokens higher priority during the encoding process. Consequently, LBPE smooths the frequency differences between short and long tokens, and thus mitigates the learning imbalance. Extensive experiments across diverse language modeling tasks demonstrate that LBPE consistently outperforms the original BPE, well demonstrating its effectiveness.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。