重评自修正掩码对扩散语言模型的纠错效果,发现其收益依赖解码设置。
Re-evaluating Confidence Remasking in Masked Diffusion Language Models
- 用置信度后处理实现无需训练的自修正机制
- 短块长解码下纠错效果微弱,与仅置信度解码接近
- 非贪婪解码时加剧多样性坍塌,适合特定场景研究
掩码扩散语言模型(dLLMs)作为自回归模型的替代方案,可通过并行生成实现更快推理。但其缺陷在于一旦标记解码便无法修正,易受早期采样错误影响。为此,部分工作提出基于置信度的后处理自修正方法,无需训练且初期表现良好。本文重新评估代表性方法WINO,在标准解码设置(较短块长)下发现其性能几乎等同于仅使用置信度解码的方法[2025]。扩展至非贪婪解码时,虽然可缓解随机性带来的错误,却加剧了已有研究中报告的多样性坍塌问题。结果表明,置信度后处理自修正的收益高度依赖具体解码配置,强调需建立更全面的评估框架。
原文摘要 · Abstract (English)
Masked diffusion language models (dLLMs) have recently emerged as a competitive alternative to autoregressive language models, with the promise of faster inference via parallel token generation. A notable limitation of the masked formulation, however, is that once a token has been unmasked it can no longer be revised, leaving dLLMs vulnerable to early sampling mistakes. To address this, a growing body of work has sought to extend masked dLLMs with self-correcting (remasking) capabilities. One appealing subset of these methods does so in a training-free, post-hoc manner based on token confidences, with encouraging early reported results. In this work, we revisit the empirical evaluation of a representative post-hoc remasking method, WINO [Hong et al., 2026], and find that under standard decoding settings (shorter block lengths) it brings little-to-no benefit over confidence-based unmasking alone [Wu et al., 2025]. Extending the evaluation to non-greedy decoding, we find that while confidence-based remasking can mitigate errors introduced by increased stochasticity to some extent, it also exacerbates the diversity collapse previously reported for confidence-based unmasking. Overall, our results show that the benefits of post-hoc confidence-based remasking are highly setting-dependent, underscoring the need for a more comprehensive evaluation framework.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。