让词频分布符合齐普夫定律，能最大化预训练模型性能。
Pre-trained Models Perform the Best When Token Distributions Follow Zipf's Law
- 通过齐普夫定律分析词频分布，指导词汇表大小选择。
- 词频越接近齐普夫分布，模型在下游任务表现越优。
- 适用于NLP、基因组学和化学等多个序列建模领域。
分词是自然语言处理及其他序列建模领域的基础步骤，词汇表大小的选择对模型性能有显著影响。然而，最优词汇表大小的确定仍缺乏系统方法，通常依赖启发式或数据集特定选择。本文提出一种基于齐普夫定律的词汇表大小确定方法，通过分析词频分布的幂律特性，发现下游任务性能与词频分布遵循幂律的程度密切相关。当词频分布紧密契合齐普夫规律时，模型在效率和效果上均达到最优。在NLP、基因组学和化学领域的大量实验表明，模型在词频分布最符合齐普夫定律时性能最佳，验证了齐普夫对齐作为词汇表大小选择的通用且稳健准则。
原文摘要 · Abstract (English)
Tokenization is a fundamental step in natural language processing (NLP) and other sequence modeling domains, where the choice of vocabulary size significantly impacts model performance. Despite its importance, selecting an optimal vocabulary size remains underexplored, typically relying on heuristics or dataset-specific choices. In this work, we propose a principled method for determining the vocabulary size by analyzing token frequency distributions through Zipf's law. We show that downstream task performance correlates with how closely token distributions follow power-law behavior, and that aligning with Zipfian scaling improves both model efficiency and effectiveness. Extensive experiments across NLP, genomics, and chemistry demonstrate that models consistently achieve peak performance when the token distribution closely adheres to Zipf's law, establishing Zipfian alignment as a robust and generalizable criterion for vocabulary size selection.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。