发现离散扩散语言模型采样器存在分布偏差，即使完美去噪也未必正确。
Is Your Diffusion Sampler Actually Correct? A Sampler-Centric Evaluation of Discrete Diffusion Language Models
- 用真实马尔可夫链构造理想去噪器，隔离采样过程误差
- 少步数采样仍存在分布不匹配，需接近序列长度才收敛
- 传统指标如困惑度无法反映采样是否真正正确
离散扩散语言模型（dLLMs）通过迭代去噪实现并行生成，是自回归模型的快速灵活替代。然而其评估困难：现有指标混淆了去噪器近似误差与采样器带来的动态误差，而自回归模型的采样天然反映学习到的概率分布。本文提出采样器中心的真值框架，将学习到的去噪器替换为基于真实马尔可夫链推导的精确隐马尔可夫后验，从而在受控环境下分离采样器误差。结果表明，即使使用真值去噪器，少步数的离散扩散采样器仍存在分布偏差，仅当采样步数趋近序列长度时过渡级误差才消失。此外，负对数似然（NLL）、生成困惑度（GenPPL）或MAUVE的提升并不意味着采样正确。代码已公开于 https://luhantang.github.io/dllm_sampler。
原文摘要 · Abstract (English)
Discrete diffusion language models (dLLMs) provide a fast and flexible alternative to autoregressive models (ARMs) via iterative denoising with parallel updates. However, their evaluation is challenging: existing metrics conflate denoiser approximation error with sampler-induced error from the sampling dynamics, a problem that does not arise for ARMs whose autoregressive sampling exactly reflects the learned probability model. We introduce a sampler-centric oracle framework that replaces learned denoisers with an exact Hidden Markov Model posterior derived from a ground-truth Markov chain, isolating sampler-induced error in a controlled setting. We show that few-step discrete diffusion samplers are not distributionally correct even under an oracle denoiser, with transition-level mismatch that vanishes only as the number of steps approaches the sequence length. Moreover, improvements in negative log-likelihood (NLL), generative perplexity (GenPPL), or MAUVE do not imply correct sampling. Code is available at https://luhantang.github.io/dllm_sampler
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。