让扩散语言模型更快更准，一次生成就能完成推理
CD4LM: Consistency Distillation and aDaptive Decoding for Diffusion Language Models
- 用一致性蒸馏让模型在不同噪声下都输出一致结果
- 动态根据置信度跳步，速度提升5.18倍且不丢质量
- 适合需要快速生成的代码和数学任务场景
自回归大语言模型在众多基准上表现优异，但解码受限于逐词生成的延迟。扩散语言模型（DLM）虽可并行生成，却存在训练与推理的根本矛盾：训练优化固定时序下的局部转移，而高效推理需对未见状态进行自适应‘长跳’修正。本文提出CD4LM框架，通过离散空间一致性蒸馏（DSCD）解耦训练与推理，并结合置信度自适应解码（CAD）。DSCD使学生模型对多样噪声状态保持轨迹不变，直接映射到干净分布；由此带来的内在鲁棒性支持CAD动态分配计算资源，大幅跳步而不损失质量。在GSM8K上，CD4LM以5.18倍的墙钟速度超越LLaDA基线；在代码与数学基准上，全面优于现有准确率-效率权衡，平均提速3.62倍且提升平均准确率。代码已开源。
原文摘要 · Abstract (English)
Autoregressive large language models achieve strong results on many benchmarks, but decoding remains fundamentally latency-limited by sequential dependence on previously generated tokens. Diffusion language models (DLMs) promise parallel generation but suffer from a fundamental static-to-dynamic misalignment: Training optimizes local transitions under fixed schedules, whereas efficient inference requires adaptive "long-jump" refinements through unseen states. Our goal is to enable highly parallel decoding for DLMs with low number of function evaluations while preserving generation quality. To achieve this, we propose CD4LM, a framework that decouples training from inference via Discrete-Space Consistency Distillation (DSCD) and Confidence-Adaptive Decoding (CAD). Unlike standard objectives, DSCD trains a student to be trajectory-invariant, mapping diverse noisy states directly to the clean distribution. This intrinsic robustness enables CAD to dynamically allocate compute resources based on token confidence, aggressively skipping steps without the quality collapse typical of heuristic acceleration. On GSM8K, CD4LM matches the LLaDA baseline with a 5.18x wall-clock speedup; across code and math benchmarks, it strictly dominates the accuracy-efficiency Pareto frontier, achieving a 3.62x mean speedup while improving average accuracy. Code is available at https://github.com/yihao-liang/CDLM
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。