通过自对比机制聚焦高信息密度词,提升扩散语言模型生成质量与速度。
Focus on the Core: Empowering Diffusion Large Language Models by Self-Contrast

- 利用高信息密度词做自对比,引导生成过程
- 在HumanEval上pass@1提升至42.68,推理速度加快58.4%
- 无需训练,适配数学、代码等复杂推理任务
扩散语言模型(DLMs)的迭代去噪机制使其具备全局上下文建模优势,但现有解码策略未能充分利用这一能力,常表现出局部偏好,忽略上下文中的信息密度差异,导致生成质量下降。我们系统研究了高信息密度(HD)词,发现:(1) 显式依赖HD词可显著提升输出质量;(2) HD词具有早期收敛倾向,比周围词更早稳定。基于此,提出训练无关的解码策略FoCore,通过将HD词临时遮蔽为负样本进行自对比,引导生成。进一步提出FoCore_A,当检测到HD词收敛后,在局部窗口内并行解码稳定候选,大幅加速。在数学、代码与逻辑推理基准测试中,FoCore持续提升生成质量与效率。例如,在HumanEval上,pass@1从39.02提升至42.68,相比标准分类器自由引导;FoCore_A将解码步数减少2.07倍,单样本延迟从20.76秒降至8.64秒(-58.4%)。
原文摘要 · Abstract (English)
The iterative denoising paradigm of Diffusion Large Language Models (DLMs) endows them with a distinct advantage in global context modeling. However, current decoding strategies fail to leverage this capability, typically exhibiting a local preference that overlooks the heterogeneous information density within the context, ultimately degrading generation quality. To address this limitation, we systematically investigate high-information-density (HD) tokens and present two key findings: (1) explicitly conditioning on HD tokens substantially improves output quality; and (2) HD tokens exhibit an early-decoding tendency, converging earlier than surrounding tokens. Motivated by these findings, we propose Focus on the Core \textbf{(FoCore)}, a training-free decoding strategy that utilizes HD tokens in a self-contrast manner, wherein HD tokens are temporarily remasked as negative samples, to guide generation. We further introduce FoCore\_Accelerate \textbf{(FoCore\_A)}, an efficient variant that, upon detecting HD token convergence, performs parallel decoding over stable candidates within a local context window, substantially accelerating generation. Extensive experiments on math, code and logical reasoning benchmarks demonstrate that FoCore consistently improves generation quality and efficiency across both LLaDA and Dream backbones. For instance, on HumanEval, FoCore improves pass@1 from 39.02 to 42.68 over standard Classifier-Free Guidance, while FoCore-A reduces the number of decoding steps by 2.07x and per-sample latency from 20.76s to 8.64s (-58.4\%).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。