通过潜空间精炼提升扩散语言模型生成速度与准确率
Latent Refinement Decoding: Enhancing Diffusion-Based Language Models by Refining Belief States
- 分两阶段：先保留未定位置的分布混合，建立全局一致信念
- 实验显示代码与推理任务准确率提升，最高提速10.6倍
- 适合追求高效并行生成的开发者与研究者
自回归模型仍是自然语言生成的标准，但因严格顺序解码导致延迟高。近期受扩散启发的方法（如LlaDA和Dream）虽实现并行生成，却存在信息丢失问题——每步丢弃非终态标记的预测分布，以及过早决策问题——局部选择缺乏全局协调。本文提出潜空间精炼解码（LRD），采用两阶段框架：第一阶段将未定位置保持为预测标记与掩码嵌入的分布混合，使模型建立更全局一致的信念；第二阶段逐步确定高置信度标记，同时保留不确定部分进行迭代反馈。KL散度动态提供可靠收敛与提前停止准则。在编码（HumanEval +6.3，MBPP +2.6）和推理（GSM8K +2.9，MATH500 +3.8）任务上的实验表明，LRD在提升准确率的同时实现最高10.6倍加速，是并行序列生成的强大且通用替代方案。
原文摘要 · Abstract (English)
Autoregressive (AR) models remain the standard for natural language generation but still suffer from high latency due to strictly sequential decoding. Recent diffusion-inspired approaches, such as LlaDA and Dream, mitigate this by generating in parallel, yet they suffer from two core limitations: information loss, as predictive distributions for non-finalized tokens are discarded at each step, and premature commitment, where local decisions are made without sufficient global coordination. We introduce Latent Refinement Decoding (LRD), a two-stage framework with Latent Refinement and a Predictive Feedback Loop. The first stage maintains masked positions as distributional mixtures of predicted tokens and the mask embedding, allowing the model to establish more globally consistent beliefs. The second stage progressively finalizes confident tokens while retaining uncertain ones for iterative feedback. KL-divergence dynamics provide a principled and reliable criterion for convergence and early stopping. Experiments across coding (HumanEval +6.3, MBPP +2.6) and reasoning (GSM8K +2.9, MATH500 +3.8) show that LRD improves accuracy while delivering speedups of up to 10.6x, making it a strong and versatile alternative for parallel sequence generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。