用KL散度让大模型学会新术语，即使分词方式不同也能迁移知识。
Vocabulary Expansion of Large Language Models via Kullback-Leibler-Based Self-Distillation
- 通过KL散度实现跨分词体系的知识蒸馏，解决词汇扩展难题。
- 在2000个代码生成任务中表现最优，显著优于传统交叉熵训练。
- 揭示新词嵌入的表征学习机制，适合关注模型可解释性的研究者。
大型预训练语言模型在微调时难以融入新领域的专有名词，尤其当训练数据量较小时。本文提出一种基于KL散度的数学严谨方法，实现冻结大模型的词汇扩展，即使教师模型与学生模型使用不同的分词方案也能完成知识蒸馏，使学生模型继承教师模型的分布知识。我们对比了该方法与传统交叉熵训练的效果，评估了多种新词嵌入初始化策略，并在嵌入初始化后进一步微调以融合新词汇。所有训练模型均在约2000个代码生成任务上进行基准测试，结果表明本方法在各项指标上均表现最佳。最后，通过机制可解释性分析，探讨了模型如何学习新词的表示，解释了性能提升原因，并揭示了词汇扩展过程中嵌入空间的结构特性。
原文摘要 · Abstract (English)
Large pre-trained language models often struggle to incorporate new domain-specific terminology when fine-tuned on small, specialized corpora. In this work, we address the challenge of vocabulary expansion in frozen LLMs by introducing a mathematically grounded method for knowledge distillation via KL divergence, even when the original and extended models use different tokenizations. This allows the student model to inherit distributional knowledge from the teacher despite differing vocabularies. We compare our KL-based distillation approach to conventional cross-entropy training, evaluating both methods across multiple strategies for initializing new token embeddings. After embedding initialization, models are further fine-tuned to integrate the new vocabulary. Each trained model is benchmarked on approximately 2000 code-generation tasks, where our approach achieves the best performance across the board. Finally, through mechanistic interpretability, we analyze how models learn representations for the new tokens, providing an explanation for the observed gains and offering insight into the structure of embedding space during vocabulary expansion.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。