清理BPE分词器中的冗余低频词，提升效率与鲁棒性。
LiteToken: Removing Intermediate Merge Residues From BPE Tokenizers
- 通过识别并移除分词过程中高频合并但实际使用极少的残留词。
- 减少分词碎片化，参数量下降10%-15%，对噪声输入更鲁棒。
- 无需微调即可适配预训练模型，适合部署优化场景。
分词是语言模型表示和处理文本的基础，但广泛使用的BPE分词器行为远未得到与其模型架构和训练同等重视的研究。本文系统分析了BPE词汇表中常见的中间合并残留：这些词在合并学习阶段频繁出现而被保留，但在实际分词时大多被进一步合并，极少被输出。这类低频词不仅浪费词汇容量，还增加对对抗性或异常输入的脆弱性。我们对常用分词器进行了实证研究，并提出LiteToken方法，可有效移除这些残留词。由于受影响的词极少使用，预训练模型通常无需额外微调即可适配新分词器。实验表明，LiteToken能降低分词碎片化，减少10%-15%参数量，同时增强对噪声或拼写错误输入的鲁棒性，整体性能保持不变。
原文摘要 · Abstract (English)
Tokenization is fundamental to how language models represent and process text, yet the behavior of widely used BPE tokenizers has received far less study than model architectures and training. In this paper, we investigate intermediate merge residues in BPE vocabularies: tokens that are frequent during merge learning so that retained in the final vocabulary, but are mostly further merged and rarely emitted when tokenizing the corpus during tokenizer usage. Such low-frequency tokens not only waste vocabulary capacity but also increase vulnerability to adversarial or atypical inputs. We present a systematic empirical characterization of this phenomenon across commonly used tokenizers and introduce LiteToken, a simple method for removing residue tokens. Because the affected tokens are rarely used, pretrained models can often accommodate the modified tokenizer without additional fine-tuning. Experiments show that LiteToken reduces token fragmentation, reduces parameters, and improves robustness to noisy or misspelled inputs, while preserving overall performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。