让旧分词器学会新词，高效适配预训练模型
Teaching Old Tokenizers New Words: Efficient Tokenizer Adaptation for Pre-trained Models
- 用持续BPE训练扩展分词器词汇，避免新增词不可达
- 跨语言多模型实验显示分词效率提升，新增词汇利用率更高
- 支持按叶子节点剪枝，保持模型质量的同时精简词表
分词器适配在将预训练语言模型迁移到新领域或新语言中起着关键作用。本文针对词汇扩展与修剪两个互补问题提出解决方案：现有扩展方法在特定领域文本上训练新分词器并添加原有词表外的词，常导致大量无法访问或从未使用的词。我们提出持续BPE训练，通过在新数据上继续预训练分词器的BPE合并过程实现词表扩展，实验表明该方法提升了分词效率并更好利用新增词汇。同时引入基于叶子节点的词表剪枝策略，在不损失模型性能前提下去除冗余词。两种方法共同构成可控词表修改的实用工具，已开源发布。
原文摘要 · Abstract (English)
Tokenizer adaptation plays an important role in adapting pre-trained language models to new domains or languages. In this work, we address two complementary aspects of this process: vocabulary extension and pruning. The common approach to extension trains a new tokenizer on domain-specific text and appends the tokens that do not overlap with the existing vocabulary, which often results in many tokens that are unreachable or never used. We propose continued BPE training that extends a pre-trained tokenizer by continuing the BPE merge learning process on new data. Experiments across multiple languages and model families show that this approach improves tokenization efficiency and leads to better utilization of added vocabulary. We also introduce leaf-based vocabulary pruning, which removes redundant tokens while preserving model quality. Together, these methods provide practical tools for controlled vocabulary modification, which we release as an open-source toolkit.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。