解决跨分词器知识蒸馏的对齐难题，提升模型压缩效果
Enhancing Cross-Tokenizer Knowledge Distillation with Contextual Dynamical Mapping
- 用上下文动态映射增强序列和词汇对齐
- 在5个模型家族中实现显著性能提升
- 适合需要跨架构模型压缩的研究者
知识蒸馏（KD）是模型压缩的重要技术，但传统方法多针对同构架构与相同分词器，难以应用于跨架构场景。跨分词器蒸馏面临两大挑战：分词策略差异导致的序列错位，以及词汇表大小与组成不匹配。现有概率匹配方法在序列与词汇层面的对齐效果有限。为此，本文提出上下文动态映射（CDM）框架，利用上下文信息提升序列对齐精度，并动态优化词汇映射。我们在五个主流模型家族（LLama3、Phi3、Gemma2、OPT、Qwen2）上构建三组教师-学生配对进行评估。结果表明，该方法在指令遵循、代码生成和数学推理等多任务基准上均显著优于现有跨分词器蒸馏基线。分析显示，结合同分词器与跨分词器蒸馏通过CDM可进一步提升性能。代码已开源。
原文摘要 · Abstract (English)
Knowledge Distillation (KD) has emerged as a prominent technique for model compression. However, conventional KD approaches primarily focus on homogeneous architectures with identical tokenizers, constraining their applicability in cross-architecture scenarios. As for the cross-tokenizer KD, the differences in the tokenizers give rise to two fundamental challenges: (1) sequence misalignment caused by divergent tokenization strategies, and (2) mismatched vocabulary size and composition. While existing probability-matching methods attempt to address these issues, their efficacy remains limited due to suboptimal alignment in both the sequence and vocabulary aspects. To overcome these limitations, we propose Contextual Dynamic Mapping (CDM), a novel cross-tokenizer distillation framework that employs contextual information to enhance sequence alignment precision and dynamically improves vocabulary mapping. We evaluated the effectiveness of our approach across five advanced and widely-used model families (i.e, LLama3, Phi3, Gemma2, OPT and Qwen2), which were configured into three distinct teacher-student pairs. Our method shows significant advantages over existing cross-tokenizer distillation baselines across diverse benchmarks, including instruction-following, code generation and math. Notably, our analysis reveals that combining conventional same-tokenizer distillation and cross-tokenizer distillation through CDM yields further performance improvements. The code is available at https://github.com/pppa2019/ContexualDynamicMapping
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。