arXiv:2508.06621cs.CL2025-08EMNLP被引 1

去掉词表合并列表也能高效分词,模型性能几乎不受影响。

Train It and Forget It: Merge Lists are Unnecessary for BPE Inference in Language Models

  • 不依赖合并列表的新型分词方法,直接压缩文本
  • 实测多种任务下模型表现下降极小,优于预期
  • 适合注重隐私保护与简化流程的场景

标准字节对编码(BPE)通过学习词汇表和详细合并列表来压缩文本。近期研究发现,该合并列表可能暴露语言模型训练数据的敏感信息。本文探讨了完全不依赖合并列表的BPE推理算法对下游任务的影响。研究涵盖两类非标准方案:一是针对合并列表的随机打乱或删减等有目的偏差;二是不依赖合并列表、仅追求文本压缩的贪婪或精确分词算法。在问答、机器翻译和开放生成等多个语言建模任务上的实验表明,虽然有目的偏离会导致模型性能显著下降,但无目标的免合并列表推理方法对下游性能影响微乎其微,通常远低于预期。这一发现为更简单且更注重隐私保护的分词方案提供了可能。

原文摘要 · Abstract (English)

Standard Byte-Pair Encoding (BPE) tokenization compresses text by pairing a learned token vocabulary with a detailed merge list. Recent work has shown that this merge list exposes a potential attack surface for extracting information about language model's training data. In this paper, we explore the downstream impact of BPE inference algorithms that do not rely on this merge list at all, and hence differ from the encoding process during BPE training. To address this question, we investigate two broad classes of BPE inference schemes that differ from BPE application during training: a) targeted deviation from merge-lists including random merge orders, and various corruptions of merge list involving deletion/truncation, and b) non-targeted BPE inference algorithms that do not depend on the merge list but focus on compressing the text either greedily or exactly. Extensive experiments across diverse language modeling tasks like accuracy-based QA benchmarks, machine translation, and open-ended generation reveal that while targeted deviation from the merge lists exhibits significant degradation in language model performance, the non-targeted merge-list-free inference algorithms result in minimal impact on downstream performance that is often much smaller than expected. These findings pave way for simpler and potentially more privacy-preserving tokenization schemes that do not catastrophically compromise model performance.

分词方法隐私保护语言模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。