通过优化分词对齐，让大模型在多语言下更高效、更快恢复性能。
TokAlign++: Advancing Vocabulary Adaptation via Better Token Alignment
- 将源与目标词汇视为双语，从单语分词表示中学习更好对齐词典。
- 仅用1000步微调即可恢复原始模型性能，多语言压缩率显著提升。
- 适合需快速适配新词汇或做分词级知识迁移的场景。
分词是大语言模型文本处理的基础步骤，文本需先被转化为标记ID才能输入模型。低效分词会导致标记序列过长，从而拖慢训练与推理速度，且阻碍如标记级蒸馏等细粒度知识迁移。为此，本文提出TokAlign++方法，通过学习更优的分词对齐词典来提升词汇自适应性能。将源与目标词汇视为两种不同语言，从单语标记表示中学习双语分词对齐词典，再按此词典重排模型参数并渐进式微调以实现适配。15种语言的实验表明，该方法显著提升了多语言文本压缩率，并保留了原始模型的大部分多语言能力。仅需1000步微调即可恢复原始模型性能；在统一词汇后，标记级蒸馏仅用235M标记即显著提升基础模型表现。
原文摘要 · Abstract (English)
Tokenization is a foundational step in the text process of Large Language Models (LLMs). Texts must be first tokenized into token IDs, which are then input to LLMs. Inefficient tokenization results in long token-ID sequences and will slow down the training and inference of LLMs. The fine-grained knowledge transfer between LLMs, like token-level distillation, is also impeded by the mismatch in vocabulary. To bridge this gap, we introduce a method named TokAlign++ to improve vocabulary adaptation performance by learning better token alignment lexicon. The source and target vocabularies are taken as two different languages, and the bilingual token alignment lexicon is learned from monolingual token representations. Model parameters are rearranged following this bilingual lexicon for new vocabulary, and progressively fine-tuned for adaptation. Experimental results on 15 languages show that our method boosts the multilingual text compression rates and preserves most of the multilingual ability of vanilla models. It costs as few as 1k steps to restore the performance of the vanilla model. After unifying vocabularies between vanilla models, token-level distillation remarkably improves the base model with only 235M tokens.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。