让密集教师指导稀疏专家模型，不依赖显式路由就能高效学习。
PADD: Path-Aligned Decompression Distillation for Non-Router Teacher to Guide MoE Student Learning
- 通过神经元聚类和专家预热，分阶段构建学生专家的多样化能力。
- 在相同计算开销下，学生模型性能超越基线并接近甚至超过教师。
- 适合想用小模型实现大模型效果的研究者或工程师。
随着大语言模型持续扩展，固定计算预算下提升模型容量愈发困难。我们提出路径对齐解压缩蒸馏（PADD），一种无需显式路由即可从密集教师模型向混合专家（MoE）学生模型进行知识蒸馏的框架，同时学习高质量路由策略。PADD将知识蒸馏分为两个阶段共四个阶段：初始化阶段（第I阶段）通过教师神经元聚类和学生专家预热，构建学生专家的多样化功能；训练阶段（第II–IV阶段）在单一训练流程中整合在线自适应蒸馏、路径优化的策略优化和奖励增强的负载均衡。在数学推理基准上的实验表明，PADD在相同推理成本下显著优于强基线，且MoE学生模型可达到甚至超越其密集教师的性能，同时实现有效知识迁移与稳定路由行为。
原文摘要 · Abstract (English)
As large language models (LLMs) continue to scale, it becomes increasingly challenging to grow model capacity under fixed computation budgets. We propose Path-Aligned Decompression Distillation (PADD), a framework for distilling knowledge from dense teachers without explicit routing into mixture-of-experts (MoE) students while learning high-quality routing policies. PADD organizes knowledge distillation into four stages in two phases: an initialization phase (Stage I) that builds diverse functionality in the student's experts through teacher neuron clustering and student-expert warmup, and a training phase (Stages II--IV) that integrates online adaptive distillation, path-refined policy optimization, and reward-augmented load balancing in a single training pipeline. Experiments on mathematical reasoning benchmarks demonstrate that PADD yields substantial gains over strong baselines at the same inference cost and that the MoE student can match or surpass its dense teacher. They also demonstrate effective teacher-to-student knowledge distillation and stable routing behavior.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。