跨分词器知识蒸馏新方法，提升小模型推理能力
ACTD: Anchor-Based Cross-Tokenizer Distillation with Residual Regularization
- 用锚点对齐不同分词器的词汇与序列结构
- 引入残差正则化降低对齐噪声，性能超越现有方法
- 支持多教师蒸馏，适合轻量化模型部署场景
知识蒸馏能将大语言模型的推理能力迁移到轻量级学生模型。为实现跨异构模型家族的知识迁移，研究者逐渐探索跨分词器蒸馏。然而，由于词汇和序列不对齐，跨分词器蒸馏仍具挑战性；而近似对齐可能引入额外噪声。为此，我们提出锚点引导的跨分词器蒸馏结合残差正则化（ACTD）。ACTD通过词汇与序列对齐弥合结构差异，同时利用新型锚点损失与残差正则化缓解对齐噪声。我们进一步将该框架扩展至多教师设置。在五个推理基准上，使用三种不同教师模型评估，ACTD达到当前最优性能。其多教师版本更优于最强单教师与多教师基线，进一步验证了方法的鲁棒性。
原文摘要 · Abstract (English)
Knowledge distillation effectively transfers reasoning capabilities from large language models to lightweight student models. To enable knowledge transfer across disparate model families, researchers increasingly explore cross-tokenizer distillation. However, cross-tokenizer distillation remains challenging due to vocabulary and sequence misalignment, while approximate vocabulary alignment can introduce additional noise into distillation. To address these challenges, we propose Anchor-Based Cross-Tokenizer Distillation with Residual Regularization (ACTD). ACTD bridges structural heterogeneity through vocabulary and sequence alignment, while mitigating alignment noise via a novel anchor loss with residual regularization. We further extend this framework to a multi-teacher setting. Evaluated across five reasoning benchmarks with three distinct teacher models, ACTD achieves state-of-the-art performance. Moreover, its multi-teacher extension outperforms the strongest single-teacher and multi-teacher baselines, further demonstrating the robustness of our method.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。