用统计显著性替代频率，提升语言模型分词效果
Significance-Gain Pair Encoding for LLMs: A Statistical Alternative to Frequency-Based Subword Merging
- 以独立性假设下的z统计量衡量词元组合的真正关联性
- 在维基文本上使困惑度降低12%-13%，每字符比特数改善0.9%-1.0%
- 适合关注分词质量与模型效率的系统设计者
子词分词是现代语言模型（包括大语言模型）的关键设计之一，字节级和字符级BPE是广泛使用的基准方法。标准BPE通过原始词元对频率选择合并，虽利于压缩，但会将高边际频次带来的高频误判为真实相邻性。本文提出显著性-增益BPE，一种可直接替换的合并准则：在独立性零假设下使用z统计量度量组合凝聚力，并结合显式的压缩感知增益项。在WikiText-103（原始字符切片）上，使用小型因果Transformer语言模型进行评估，报告了依赖词元的困惑度及与分词器无关的每字符比特数（BPC）。在典型运行点，显著性-增益BPE使验证集和测试集困惑度分别降低13%和12%，验证集和测试集BPC改善约0.9%至1.0%。词汇量扫面进一步显示，在多数相近压缩率对比中，其BPC更低，表明基于统计的合并选择可在多种压缩条件下提升单位文本的预测效率。
原文摘要 · Abstract (English)
Subword tokenization is a key design choice for modern language models, including large language models (LLMs), with byte- and character-level BPE serving as a widely used baseline. Standard BPE selects merges by raw pair frequency, which favors compression but can conflate true adjacency cohesion with pairs that are frequent due to high marginal counts. This paper introduces Significance-Gain BPE, a drop-in alternative merge criterion that measures cohesion via a z-statistic under an independence null model and combines it with an explicit compression-aware gain term. Significance-Gain BPE is evaluated on WikiText-103 (raw) character slices using a small causal Transformer language model, reporting both token-dependent perplexity and the tokenizer-invariant metric bits per character (BPC). At a representative operating point, Significance-Gain BPE reduces validation and test perplexity by 13% and 12%, respectively, and improves validation and test BPC by about 0.9 to 1.0%. A vocabulary-size sweep further shows lower BPC in most closest-compression comparisons, suggesting that statistically grounded merge selection can improve predictive efficiency per unit of raw text across a range of compression regimes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。