为特定领域定制更高效的分词器，提升语言模型压缩与推理效率。
AdaptBPE: From General Purpose to Specialized Tokenizers
- 基于适配语料频次，替换低效词元，优化词表结构。
- 相同词表规模下，测试文本压缩率优于基线方法。
- 轻量级适配，适合需要高效分词的垂直领域或任务。
子词分词方法（如字节对编码，BPE）显著影响大语言模型的性能与效率。传统做法是训练一个通用分词器，在训练和推理中统一处理所有文本。然而，通用词元集在特定领域或语言上可能造成效率损失。为此，我们提出一种后训练适配策略：根据适配语料中的词元频率，选择性替换低效词元。该算法针对给定目标词表大小，找出最有效编码适配语料的词元集合。在多语言生成与分类任务上的大量实验表明，所生成的适配分词器在相同词表规模下，能更有效地压缩测试语料。该方法相当于词汇层面的微调，是一种轻量级适配机制，可为特定领域或任务实现优化分词。代码与数据已公开于 https://github.com/vijini/Adapt-BPE.git。
原文摘要 · Abstract (English)
Subword tokenization methods, such as Byte-Pair Encoding (BPE), significantly impact the performance and efficiency of large language models (LLMs). The standard approach involves training a general-purpose tokenizer that uniformly processes all textual data during both training and inference. However, the use of a generic set of tokens can incur inefficiencies when applying the model to specific domains or languages. To address this limitation, we propose a post-training adaptation strategy that selectively replaces low-utility tokens with more relevant ones based on their frequency in an adaptation corpus. Our algorithm identifies the token inventory that most effectively encodes the adaptation corpus for a given target vocabulary size. Extensive experiments on generation and classification tasks across multiple languages demonstrate that our adapted tokenizers compress test corpora more effectively than baselines using the same vocabulary size. This method serves as a lightweight adaptation mechanism, akin to a vocabulary fine-tuning process, enabling optimized tokenization for specific domains or tasks. Our code and data are available at https://github.com/vijini/Adapt-BPE.git.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。