解决大模型蒸馏中分词器不匹配导致的性能下降问题
Dual-Space Knowledge Distillation with Key-Query Matching for Large Language Models with Vocabulary Mismatch
- 通过生成对抗学习优化跨模型键值匹配
- 在分布外数据上文本生成质量提升0.37（ROUGE-L）
- 适合需要跨分词器高效部署大模型的场景
大语言模型（LLM）在各类语言任务中表现优异，但因其规模庞大而难以部署。知识蒸馏（KD）通过训练小型学生模型模仿大型教师模型，在保持性能的同时提升效率。针对具有不同分词器的模型间蒸馏，双空间知识蒸馏结合跨模型注意力（DSKD-CMA）已达到先进水平，但其内部机制仍不清晰。本文通过人工标记对齐与热力图可视化系统分析了DSKD-CMA的注意力机制，揭示其优缺点。基于此，提出新方法DSKD-CMA-GA，采用生成对抗学习缓解由不同模型计算出的键与查询之间的分布不匹配问题。实验表明，该方法在文本生成质量上带来适度但稳定的提升，尤其在分布外数据上平均提升ROUGE-L达0.37，缩小了跨分词器与同分词器蒸馏间的差距。
原文摘要 · Abstract (English)
Large language models (LLMs) achieve state-of-the-art (SOTA) performance across language tasks, but are costly to deploy due to their size and resource demands. Knowledge Distillation (KD) addresses this by training smaller Student models to mimic larger Teacher models, improving efficiency without significant performance loss. Dual-Space Knowledge Distillation with Cross-Model Attention (DSKD-CMA) has emerged as a SOTA method for KD between LLMs with distinct tokenizers, yet its internal workings remain largely opaque. In this work, we systematically analyse the attention mechanism of DSKD-CMA through manual token alignment probing and heatmap visualisations, revealing both strengths and limitations. Building on this, we introduce a novel method, DSKD-CMA-GA, based on Generative Adversarial (GA) learning, to address the mismatched distributions between the keys and queries computed from distinct models. Experiments show modest but consistent ROUGE-L gains in text generation quality, particularly on out-of-distribution data (+0.37 on average), narrowing the gap between cross- and same-tokenizer KD.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。