arXiv:2606.16847cs.CLcs.AI2026-06

通过锚点令牌提升扩散语言模型的生成质量与速度

Follow the Latent Roadmap: Navigating Revocable Decoding for Diffusion LLMs with Anchor Tokens

论文配图:Follow the Latent Roadmap: Navigating Revocable Decoding for Diffusion LLMs with Anchor Tokens
图 1 · 摘自论文原文
  • 用时间一致性识别可信锚点,分离可靠与不确定内容
  • 提升准确率最高6.4%,推理吞吐量加速7.2倍
  • 无需训练,适合追求高效高质生成的开发者

扩散大语言模型(dLLMs)虽支持并行生成,但存在解码速度与质量的权衡。现有可撤销解码策略常在混合质量上下文中运行,导致两个关键问题:错误传播(新词吸收错误信息)和局部错误强化(错误相互巩固难以检测)。为此,我们提出ASRD(锚点监督可撤销解码),一种无需训练的嵌入空间框架。ASRD通过时间一致性识别可信锚点令牌,并构建动态锚点缓存。引入两种互补机制:(1)锚点引导生成,将加权锚点信号注入掩码位置,隐式纠正注意力聚焦于可靠全局结构;(2)锚点扰动验证,对不确定候选令牌施加正交扰动,破坏依赖脆弱局部共识的错误。在数学与编程基准上的大量实验表明,ASRD优于近期重掩码基线,在准确率上提升最高达6.4%,推理吞吐量最高加速7.2倍。

原文摘要 · Abstract (English)

Diffusion Large Language Models (dLLMs) offer a promising avenue for parallel generation but face a trade-off between decoding speed and quality. While revocable decoding strategies attempt to mitigate errors by verifying and remasking tokens, they typically operate within a mixed-quality context. This leads to two critical failures: \textit{Error Propagation}, where new tokens absorb toxic information from erroneous context, and \textit{Local Error Reinforcement}, where errors mutually reinforce each other to evade detection. To alleviate these challenges, we propose ASRD (Anchor Supervised Revocable Decoding), a training-free framework that operates within the embedding space. ASRD explicitly decouples the decoding context into trusted \textit{Anchor Tokens}, which are identified via temporal consistency, and uncertain candidates. Leveraging a dynamic Anchor Tokens Cache, we introduce two complementary mechanisms: (1) Anchor-Guided Generation, which injects entropy-weighted anchor signals into masked positions to implicitly rectify attention toward the reliable global skeleton; and (2) Anchor-Perturbed Verification, which applies orthogonal perturbations to uncertain candidate tokens, destabilizing and remasking errors driven by fragile local consensus. Extensive experiments on math and coding benchmarks demonstrate that ASRD outperforms recent remasking baselines, achieving accuracy improvements of up to 6.4\% while accelerating inference throughput by up to 7.2$\times$.

扩散模型生成优化解码加速

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。