无需训练即可迁移大模型分词器,保持性能不下降。
Training-Free Tokenizer Transplantation via Orthogonal Matching Pursuit
- 用正交匹配追踪重建未登录词嵌入,通过共享词构建稀疏组合。
- 在两个跨分词器任务上零样本性能优于现有方法,数学推理能力更稳定。
- 适合需要快速适配新分词器的场景,如模型合并、领域词表定制。
我们提出一种无需训练的方法,通过正交匹配追踪(OMP)重构预训练大语言模型中未见词的嵌入,实现分词器移植。具体分为两阶段:首先用少量共享锚点词在源嵌入空间中表示新词,再将这些稀疏系数映射回目标模型嵌入空间。在两个挑战性跨分词器任务——Llama→Mistral NeMo(12B)和Qwen→Llama(1B)上,OMP在多个基准测试中实现了最优的零样本性能保持,而其他零样本方法显著退化。相比基线方法(零初始化、均值初始化及WECHSEL、FOCUS、ZETT等),OMP始终表现最佳,有效缓解了大分词器差异带来的问题,且无需梯度更新。分析发现,数值分词方案不匹配是影响数学推理能力的关键挑战。该技术可直接复用预训练模型权重,支持跨分词器知识蒸馏、推测解码、集成、合并及特定领域词表适配。我们已将其集成至开源工具mergekit-tokensurgeon,用于事后词表重对齐。
原文摘要 · Abstract (English)
We present a training-free method to transplant tokenizers in pretrained large language models (LLMs) by reconstructing unseen token embeddings via Orthogonal Matching Pursuit (OMP). Specifically, we approximate each out-of-vocabulary token as a sparse linear combination of shared tokens, in two phases: first, compute each new token's representation in the donor embedding space with a small dictionary of shared anchor tokens, then transfer these same sparse coefficients back into the base model's embedding space. On two challenging cross-tokenizer tasks--Llama$\to$Mistral NeMo (12B) and Qwen$\to$Llama (1B)--we show that OMP achieves best zero-shot preservation of the base model's performance across multiple benchmarks, while other zero-shot approaches degrade significantly. Compared to baselines (zero-init, mean-init, and existing approaches like WECHSEL, FOCUS, ZETT), OMP consistently achieves the best overall performance, effectively bridging large tokenizer discrepancies without gradient updates. Our analysis further identifies mismatched numerical tokenization schemes as a critical challenge for preserving mathematical reasoning capabilities. This technique enables direct reuse of pretrained model weights with new tokenizers, facilitating cross-tokenizer knowledge distillation, speculative decoding, ensembling, merging, and domain-specific vocabulary adaptations. We integrate our method into the open-source mergekit-tokensurgeon tool for post hoc vocabulary realignment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。