arXiv:2510.05090cs.CLcs.AI2025-10被引 5

提出可重审已生成词元的解码方法,提升扩散型大模型输出质量。

Finish First, Perfect Later: Test-Time Token-Level Cross-Validation for Diffusion Large Language Models

  • 通过两阶段策略:填空+迭代重掩码解码,允许修正早期错误
  • 在5个基准上实现一致提升,相同算力下优于基线模型
  • 无需训练,适合追求高精度的扩散语言模型应用

扩散型大语言模型(dLLMs)作为自回归模型的有前途替代方案,具备并行加速解码和双向上下文建模的优势。然而,现有离散dLLMs的解码策略存在关键缺陷:一旦词元被接受,后续步骤中无法修改,导致早期错误持续影响中间与最终输出质量。为此,我们提出Tolerator(词元级交叉验证精炼)——一种无需训练的解码策略,利用预测词元间的交叉验证机制。不同于单一渐进式去掩码流程,Tolerator采用两阶段设计:(i) 序列填空,(ii) 迭代重掩码并解码部分词元,其余视为上下文。该设计使先前接受的词元可被重新审视与修正,从而提升扩散解码的可靠性。我们在涵盖语言理解、代码生成与数学推理的五个标准基准上评估Tolerator。实验表明,在相同计算预算下,该方法持续优于基线模型。结果表明,解码算法对释放扩散大模型潜力至关重要。代码与数据已公开。

原文摘要 · Abstract (English)

Diffusion large language models (dLLMs) have recently emerged as a promising alternative to autoregressive (AR) models, offering advantages such as accelerated parallel decoding and bidirectional context modeling. However, the vanilla decoding strategy in discrete dLLMs suffers from a critical limitation: once a token is accepted, it can no longer be revised in subsequent steps. As a result, early mistakes persist across iterations, harming both intermediate predictions and final output quality. To address this issue, we propose Tolerator (Token-Level Cross-Validation Refinement), a training-free decoding strategy that leverages cross-validation among predicted tokens. Unlike existing methods that follow a single progressive unmasking procedure, Tolerator introduces a two-stage process: (i) sequence fill-up and (ii) iterative refinement by remasking and decoding a subset of tokens while treating the remaining as context. This design enables previously accepted tokens to be reconsidered and corrected when necessary, leading to more reliable diffusion decoding outputs. We evaluate Tolerator on five standard benchmarks covering language understanding, code generation, and mathematics. Experiments show that our method achieves consistent improvements over the baselines under the same computational budget. These findings suggest that decoding algorithms are crucial to realizing the full potential of diffusion large language models. Code and data are publicly available.

扩散模型解码优化语言模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。