改进BPE分词算法，训练时自动剔除低效合并，提升模型性能。
BPE Gets Picky: Efficient Vocabulary Refinement During Tokenizer Training
- 在分词训练中动态剔除产生冗余符号的合并操作
- 消除未充分训练的分词符，提升词汇效率且不降低压缩率
- 适合追求分词质量与下游任务性能的NLP研究者
语言模型可显著受益于高效的分词机制。然而，当前仍主要采用经典BPE算法，该方法虽简单可靠，却易导致分词符训练不足和压缩效率不佳，影响下游性能。本文提出Picky BPE，一种在分词器训练过程中进行词汇优化的改进型BPE算法。该方法通过移除生成中间“垃圾”符号的合并操作，提升词汇效率，消除未充分训练的分词符，且不损害文本压缩能力。实验表明，该方法不影响下游任务性能，在多个场景下甚至有所提升。
原文摘要 · Abstract (English)
Language models can largely benefit from efficient tokenization. However, they still mostly utilize the classical BPE algorithm, a simple and reliable method. This has been shown to cause such issues as under-trained tokens and sub-optimal compression that may affect the downstream performance. We introduce Picky BPE, a modified BPE algorithm that carries out vocabulary refinement during tokenizer training. Our method improves vocabulary efficiency, eliminates under-trained tokens, and does not compromise text compression. Our experiments show that our method does not reduce the downstream performance, and in several cases improves it.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。