通过修剪低频子词提升BPE编码效率，不增加模型词汇量却能更紧凑地表示文本。
Pruned BPE: Post-training Visibility Pruning and Token Reallocation for Byte Pair Encoding
- 训练后按使用频率筛选子词，低频的仅作内部构建用
- 相同词汇量下，编码长度减少0.27%至0.36%
- 适合追求高效编码的轻量化模型部署场景
Byte Pair Encoding（BPE）广泛用于子词分词，但标准BPE将所有学习到的合并词暴露给下游模型，包括仅作为中间构造单元、极少出现在最终语料中的词。本文提出Pruned BPE，一种训练后的可见性剪枝与词元重分配方法，将合并构建过程与模型可见词汇分离。标准BPE训练完成后，根据最终暴露频率评估词元：低暴露词元保留为仅内部使用的合并节点，其可见词汇槽位重分配给在重启训练中获得更高暴露度的候选词。编码时，仅内部的词元递归展开为可见后代，同时保持原始BPE合并顺序。在两个非重叠的英文与中文主导语料及其组合上实验表明，相同训练语料、评估语料及模型可见词汇大小下，Pruned BPE持续降低编码长度。在40%暴露阈值下，同语料评估中减少约0.27%–0.36%。在共享精确最小词元动态规划编码器的词汇仅评估中，优势仍达约0.23%–0.31%，表明改进源于更高效的可见词汇。这些增益相当于无需增加2000个标准BPE词元即可实现约1.5%–3.8%的边际缩减。定性分析显示，内部仅使用词元包含可复用的英文片段、中文部件、部分UTF-8字节序列和结构化文本片段。结果表明，训练后可见性剪枝可在不增加模型词汇量的前提下提升BPE词汇效率。
原文摘要 · Abstract (English)
Byte Pair Encoding (BPE) is widely used for subword tokenization, but standard BPE exposes every learned merge token to the downstream model, including tokens that mainly serve as intermediate construction units and rarely appear in the final encoded corpus. This paper proposes Pruned BPE, a post-training visibility-pruning and token-reallocation method that separates merge construction from model-visible vocabulary selection. After standard BPE training, tokens are evaluated by final exposure. Low-exposure tokens are retained as internal-only merge nodes, while their visible vocabulary slots are reassigned to better-exposed candidates learned through resumed training. During encoding, internal-only tokens are recursively expanded into visible descendants while the original BPE merge order is preserved. Experiments on two non-overlapping English- and Chinese-dominated corpora and their combination show that Pruned BPE consistently reduces encoded length relative to Standard BPE at the same training corpus, evaluation corpus, and model-visible vocabulary size. At a 40% exposure threshold, the reduction is approximately 0.27%--0.36% on same-corpus evaluations. In a vocabulary-only evaluation using a shared exact minimum-token dynamic-programming encoder, Pruned BPE retains an advantage of approximately 0.23%--0.31%, indicating that the improvement arises from a more efficient visible vocabulary. These gains represent a meaningful fraction of the approximately 1.5%--3.8% marginal reduction that would otherwise require adding another 2K Standard BPE tokens. Qualitative analysis shows that internal-only tokens include reusable English fragments, Chinese components, partial UTF-8 byte sequences, and structured-text fragments. The results indicate that post-training visibility pruning can improve BPE vocabulary efficiency without increasing the vocabulary exposed to the language model.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。