通过原位扩展分词器,让大模型更高效支持新增语言。
In-Place Tokenizer Expansion for Pre-trained LLMs
- 在原有分词器基础上增量合并多语言语料,保持旧词不变
- 新词用旧子词组合表示,嵌入层均值初始化,性能损失小
- 对印地语、越南语等实现2.4~4倍的分词压缩,解码提速2.2~3.7倍
预训练模型的分词器在训练初期固定,其词汇量按初始语料比例分配,难以适应后续语言扩展需求。新增语言常被过度切分为更多子词,导致延迟、计算和能耗上升。云模型因嵌入和LM头占比较小可扩容,而轻量设备受限于带宽,通常使用小词汇量且不支持扩展语言。本文提出原位分词器扩展方法:在多语言语料上继续原有BPE合并过程,确保多数源词保持为单个词元,新词均由源子词精确重构。保留原有词元嵌入,新词嵌入初始化为对应子词嵌入均值。经两阶段微调(仅嵌入层训练+全模型持续预训练),恢复原始检查点性能。该方法应用于80亿参数专家混合模型LFM2-8B-A1B,生成具有12.8万词汇量的LFM2.5-8B-A1B。相较原分词器,印地语、越南语编码分别减少约2.4倍和2.6倍(泰语最高达4.0倍)。结合大词表带来的每词元开销,估计在参考设备上解码速度提升2.2至3.7倍。模型权重与扩展分词器已公开,并报告了方法设计中的负面发现。
原文摘要 · Abstract (English)
A tokenizer fixed at the start of pre-training allocates vocabulary in proportion to the pre-training corpus, reflecting the deployment priorities at that time. When those priorities shift, languages added later are split into many more tokens per word, which can raise latency, compute, and energy consumption for users of those languages. Cloud models can afford a broad vocabulary because the embedding and LM-head matrices are a small fraction of their parameters. On a compact model those matrices are a material share of per-token decode bandwidth, so on-device models ship small vocabularies and accept fragmentation outside a fixed language set. We present tokenizer expansion, an in-place recipe for upgrading a pre-trained model's tokenizer when the model producer controls its design. We continue the existing tokenizer's BPE merges on a multilingual corpus, so most source tokens carry over unchanged as single tokens and every new token has an exact decomposition into source tokens. We copy the carried-over embedding rows unchanged and initialize new rows as the mean of their source sub-token embeddings. A two-stage adaptation, embedding-only training then full-model continued pre-training, recovers source-checkpoint quality. We apply the recipe to a continued pre-trained checkpoint of LFM2-8B-A1B, an 8B-parameter Mixture-of-Experts model, to help produce LFM2.5-8B-A1B with a 128K tokenizer. The expanded tokenizer encodes Hindi and Vietnamese in roughly $2.4\times$ and $2.6\times$ fewer tokens than the source (up to $4.0\times$ on Thai). Combining these reductions with the measured per-token cost of the larger vocabulary, we estimate a $2.2$-$3.7\times$ per-character decode speedup for these languages across our reference devices. We release the model weights and the expanded tokenizer, and report the negative findings that shaped the recipe.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。