为字节级BPE分词器适配书写系统，提升低频语言效率。
Writing-System-Level Tokenizer Adaptation for Byte-Level BPE
- 通过目标可到达分解实现分词符号的精准迁移。
- 乌克兰语分词量减少33.5%~36.6%，英语和欧洲语种变化小于0.05%。
- 保留原模型词汇行78.5%以上，适合多语言部署与模型兼容场景。
预训练的字节级BPE分词器对低频语言分割效率低下。更换分词器会改变几乎每个标记的ID，而词汇扩展则增大模型嵌入和输出矩阵。本文研究一种后处理适配方法，在保持模型-词汇表大小不变的前提下，尽可能保留原有标记到ID的映射关系。直接转移语言特异性分词器的标记无法保证通过目标BPE合并图推导：插入条目可能与目标贪婪合并优先级冲突。我们将其形式化为合并顺序问题，并提出基于BPE引导的插入机制，通过目标可达分解构建每个转移标记。该流程采用脚本感知行选择以减少副作用碎片化，重构目标脚本字节级前置条件，并应用引导插入确保合并图可达性。在Nemotron和GPT-OSS的乌克兰语适配中，分词数分别降低33.5%和36.6%，英文及四语言欧洲聚合体变化低于0.05%，原模型词汇行保留率分别为78.5%和77.3%。约束匹配的全局与频率删除实现类似压缩，但使英文/欧洲语种分词量增加0.7%-2.2%；全量重训练稍优但无同ID行保留，且英文分词量增加7.6%-8.6%。重新分配导致三语言西里尔字母微聚合分词量上升6.7%/10.1%。结构审计显示，所有28,134/45,398个插入的BPE节点在普通优先级合并下均可达，且未破坏任何保留的同ID词汇行。代码与分词器已开源。
原文摘要 · Abstract (English)
Pretrained byte-level BPE tokenizers can segment underrepresented languages inefficiently. Replacing a tokenizer changes the meaning of nearly every token ID, while vocabulary expansion enlarges the model's embedding and output matrices. We study post-hoc adaptation that keeps the model-vocabulary size fixed and preserves most existing token-to-ID assignments as a construction-time compatibility property. Directly transferring tokens from a language-specific tokenizer does not guarantee derivability through the target BPE merge graph: an inserted entry can conflict with the target's greedy merge ranks. We formalize this failure as the merge ordering problem and introduce BPE-guided insertion, which builds each transferred token through a target-reachable decomposition. Our pipeline uses script-aware row selection to limit collateral fragmentation, reconstructs target-script byte-level prerequisites, and applies guided insertion to maintain merge-graph reachability. On Ukrainian adaptations of Nemotron and GPT-OSS, it reduces token counts by 33.5% and 36.6%, keeps changes on English and the evaluated four-language European aggregate within 0.05%, and retains 78.5%/77.3% of original model-vocabulary rows at the same IDs. Constraint-matched global and frequency-based removal achieve similar Ukrainian compression but increase English/European token counts by 0.7-2.2%; fresh same-size retraining compresses Ukrainian slightly more but retains effectively no same-ID rows and increases English token counts by 7.6-8.6%. The reallocation increases token counts on the evaluated three-language Cyrillic micro-aggregate by 6.7%/10.1%. Structural audits find all 28,134/45,398 inserted BPE nodes reachable under ordinary rank-ordered merging and no retained same-ID model-vocabulary entry newly broken. We release all tokenizers and code.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。