提升扩散语言模型生成质量与速度,通过动态调整采样策略实现更连贯的文本生成。
Beyond Confidence: Adaptive and Coherent Decoding for Diffusion Language Models
- 引入轨迹修正机制,利用历史上下文提升生成连贯性。
- 动态调整每步采样预算,实现3.48倍加速和3.91%性能提升。
- 适用于追求高效高质文本生成的研究者与开发者。
扩散语言模型(DLMs)因其任意顺序生成能力取得显著进展。然而,现有推理方法多依赖局部、即时步骤的度量(如置信度或熵),缺乏可靠视角,常导致采样轨迹不一致与生成质量不佳。为此,我们提出一致上下文解码(CCD),其核心创新包括:首先,采用轨迹修正机制,利用历史上下文增强序列一致性,可提前剔除次优路径;理论证明该机制等价于通过条件互信息建模历史步骤的一致性。基于此,我们进一步解决传统均匀采样预算效率低的问题,提出自适应采样策略,根据一致性度量动态调整每步未掩码预算。实验表明,该方法在Dream与LLaDA多个基准上同时提升生成质量与推理速度,最高实现3.48倍加速,性能提升达3.91%。
原文摘要 · Abstract (English)
Diffusion Language Models (DLMs) have recently achieved significant success due to their any-order generation capabilities. However, existing inference methods typically rely on local, immediate-step metrics such as confidence or entropy which inherently lack a more reliable perspective. This limitation frequently leads to inconsistent sampling trajectories and suboptimal generation quality. To address this, we propose Coherent Contextual Decoding (CCD), a novel inference framework built upon two core innovations. First, CCD employs a trajectory rectification mechanism that leverages historical context to enhance sequence coherence, enabling the early rejection of suboptimal paths. We demonstrate that this mechanism is theoretically equivalent to modeling the consistency of historical steps via the conditional mutual information between context and token predictions. Building on this theoretical insight, we further address the inefficiency of conventional uniform decoding budgets. Instead of rigid allocations based on diffusion steps, we introduce an adaptive sampling strategy that dynamically adjusts the unmasking budget for each step according to our consistency metric. Consequently, our method significantly improves the quality of generation trajectories while accelerating the sampling process. Empirically, our method achieves a simultaneous enhancement in both inference speed and performance across diverse benchmarks on Dream and LLaDA, delivering up to 3.48x speedup alongside 3.91% performance improvement.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。