发现离散扩散模型暗中实现了贝叶斯后验，无需训练即可估计不确定性。
Your Absorbing Discrete Diffusion Secretly Models the Bayesian Posterior
- 利用蒙特卡洛采样在推理时聚合多轮去噪结果，直接估计后验分布。
- 在WikiText-2上，K=128时困惑度逼近理论最优值39，误差极小。
- 适合需要可信不确定性的文本生成与模型评估场景。
离散扩散语言模型通过随机掩码输入重建文本，但在弱假设下其去噪器已精确实现原始词元的贝叶斯后验。我们证明，在前向破坏分布下的期望去噪输出可恢复真实后验，且简单的蒙特卡洛估计器以O(1/sqrt(K))速率收敛，并具有有限样本集中界。基于此，我们提出推理时集成方法：运行K次独立去噪并聚合后验均值与方差，无需额外训练。在WikiText-2上，我们的MC边际采样器在K=128时，将零样本困惑度逼近理论值约39，误差仅数点；每标记方差与重构误差呈强秩相关（Spearman rho = 0.996）。该方法按计算量比例提供校准不确定性，实现计算开销与后验保真度的直接权衡。
原文摘要 · Abstract (English)
Discrete diffusion language models learn to reconstruct text from randomly masked inputs, yet under mild assumptions their denoiser already implements the exact Bayesian posterior over the original tokens. We prove that the expected denoiser output under the forward corruption distribution recovers the true posterior, and that a simple Monte Carlo estimator converges to this posterior at rate O(1/sqrt(K)) with finite-sample concentration bounds. Building on this insight, we introduce an inference-time ensemble that runs K independent denoising passes and aggregates both posterior means and variances without any extra training. On WikiText-2, our MC-marginal sampler recovers the analytic lambda-DCE zero-shot perplexity (approximately 39) to within a few points at K=128, and its per-token variance shows a strong rank correlation with reconstruction error (Spearman rho = 0.996). This cost-proportional procedure yields calibrated uncertainty estimates and a direct trade-off between compute and posterior fidelity in discrete diffusion LMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。