首个实现扩散语言模型跨架构蒸馏的方法，让小模型高效学习大模型知识。
Turning the TIDE: Cross-Architecture Distillation for Diffusion Large Language Models
- 设计三模块框架，动态调节蒸馏强度并增强教师上下文信息。
- 0.6B小模型在8个基准上平均提升1.53分，代码生成得分达48.78。
- 适合资源受限场景下高效部署高性能扩散语言模型。
扩散大型语言模型（dLLMs）具备并行解码和双向上下文优势，但当前顶尖dLLMs需数十亿参数才能达到竞争力表现。现有蒸馏方法仅限同架构内减少推理步数，未解决教师与学生模型在架构、注意力机制和分词器上差异的跨架构知识迁移问题。本文提出TIDE，首个面向dLLM的跨架构蒸馏框架，包含三个模块：(1) TIDAL，根据训练进度与扩散时间步联合调节蒸馏强度，以应对教师模型噪声依赖的可靠性变化；(2) CompDemo，通过互补掩码分割丰富教师上下文，提升重掩码下的预测性能；(3) Reverse CALM，一种反向分块级似然匹配的跨分词器目标，实现有界梯度与双端噪声过滤。将8B稠密型和16B MoE型教师模型分别蒸馏至0.6B学生模型，通过两条异构路径，平均在8个基准上超越基线1.53分，代码生成表现显著，HumanEval得分达48.78，远超自回归基线的32.3。
原文摘要 · Abstract (English)
Diffusion large language models (dLLMs) offer parallel decoding and bidirectional context, but state-of-the-art dLLMs require billions of parameters for competitive performance. While existing distillation methods for dLLMs reduce inference steps within a single architecture, none address cross-architecture knowledge transfer, in which the teacher and student differ in architecture, attention mechanism, and tokenizer. We present TIDE, the first framework for cross-architecture dLLM distillation, comprising three modular components: (1) TIDAL, which jointly modulates distillation strength across training progress and diffusion timestep to account for the teacher's noise-dependent reliability; (2) CompDemo, which enriches the teacher's context via complementary mask splitting to improve predictions under heavy masking; and (3) Reverse CALM, a cross-tokenizer objective that inverts chunk-level likelihood matching, yielding bounded gradients and dual-end noise filtering. Distilling 8B dense and 16B MoE teachers into a 0.6B student via two heterogeneous pipelines outperforms the baseline by an average of 1.53 points across eight benchmarks, yielding notable gains in code generation, where HumanEval scores reach 48.78 compared to 32.3 for the AR baseline.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。