通过词元对齐实现高效词汇替换，提升多语言模型训练与知识迁移效率。
TokAlign: Efficient Vocabulary Adaptation via Token Alignment
- 基于词元共现关系学习词表映射，重排并微调模型参数以适配新词表。
- 初始化后困惑度降至1.2e²，较基线降低约65%，仅需5千步恢复原始性能。
- 适合需要快速适配新语言或领域、追求高效知识蒸馏的研究者与工程师。
分词是大语言模型处理文本的基础步骤。在新领域或新语言中，分词器效率低下会拖慢模型训练与生成速度，词表不匹配也阻碍了模型间如词元级蒸馏等深度知识迁移。为此，我们提出一种名为TokAlign的高效方法，从词元共现视角替换语言模型的词表，并进一步实现模型间的词元级知识迁移。该方法首先通过学习词元ID的一一映射矩阵，将源词表对齐至目标词表；随后重排模型参数（包括嵌入层），并进行渐进式微调以适应新词表。实验表明，TokAlign显著提升了多语言文本压缩率与词表初始化效果，初始化后困惑度从强基线的3.4e²降至1.2e²。跨不同参数规模模型的实验验证了其有效性与泛化能力，仅需5000步即可恢复原模型性能。词表统一后，词元级蒸馏相较句级蒸馏提升4.4%，且仅消耗2.35亿词元。
原文摘要 · Abstract (English)
Tokenization serves as a foundational step for Large Language Models (LLMs) to process text. In new domains or languages, the inefficiency of the tokenizer will slow down the training and generation of LLM. The mismatch in vocabulary also hinders deep knowledge transfer between LLMs like token-level distillation. To mitigate this gap, we propose an efficient method named TokAlign to replace the vocabulary of LLM from the token co-occurrences view, and further transfer the token-level knowledge between models. It first aligns the source vocabulary to the target one by learning a one-to-one mapping matrix for token IDs. Model parameters, including embeddings, are rearranged and progressively fine-tuned for the new vocabulary. Our method significantly improves multilingual text compression rates and vocabulary initialization for LLMs, decreasing the perplexity from 3.4$\text{e}^2$ of strong baseline methods to 1.2$\text{e}^2$ after initialization. Experimental results on models across multiple parameter scales demonstrate the effectiveness and generalization of TokAlign, which costs as few as 5k steps to restore the performance of the vanilla model. After unifying vocabularies between LLMs, token-level distillation can remarkably boost (+4.4% than sentence-level distillation) the base model, costing only 235M tokens.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。