为扩散语言模型提供轻量级不确定性评估,兼顾速度与可靠性。
Uncertainty Quantification for Large Language Diffusion Models

- 利用去噪过程中的中间生成、令牌遮蔽动态和去噪复杂度,构建零样本不确定性信号。
- 在三个任务、八个数据集上表现接近采样基线,计算开销降低100倍。
- 适合追求高效且可信生成的开发者和部署者使用。
大语言扩散模型(LLDMs)作为自回归模型的替代方案,通过更高并行性实现更快推理。与自回归大模型类似,它们仍易产生幻觉,因此可靠的不确定性量化(UQ)对安全部署至关重要。然而,现有UQ方法与这一新范式本质不匹配:要么假设自回归分解,要么依赖昂贵的重复采样,削弱了LLDM的效率优势。本文首次系统研究LLDM的不确定性量化,提出基于迭代去噪过程的轻量级、零样本不确定性信号,利用中间生成、令牌重遮蔽动态和去噪复杂度。进一步将先进的UQ方法适配至LLDM,结合掩码扩散似然与轨迹语义差异。证明期望轨迹差异下界于掩码扩散训练目标,从而将其作为不确定性得分。在三个任务、八个数据集、两个模型上的全面实验表明,该方法在成本与性能间取得优异平衡:逼近最强采样基线,同时计算开销降低高达100倍。结果表明,LLDM可同时实现快速推理与可靠幻觉检测。
原文摘要 · Abstract (English)
Large Language Diffusion Models (LLDMs) are emerging as an alternative to autoregressive models, offering faster inference through higher parallelism. Similar to autoregressive LLMs, they remain prone to hallucinations, making reliable uncertainty quantification (UQ) crucial for safe deployment. However, existing UQ methods are fundamentally misaligned with this new paradigm: they assume autoregressive factorization or use expensive repeated sampling, negating the efficiency of LLDMs. In this work, we present the first systematic study of UQ for LLDMs and propose lightweight, zero-shot uncertainty signals derived from the iterative denoising process, leveraging intermediate generations, token remasking dynamics, and denoising complexity. We further adapt a state-of-the-art UQ method to LLDMs by combining masked diffusion likelihoods with trajectory-based semantic dissimilarity. We prove that expected trajectory dissimilarity lower bounds the masked diffusion training objective, which motivates its usage as an uncertainty score. Comprehensive experiments across three tasks, eight datasets, and two models show that our method achieves a great cost-performance trade-off: it approaches the strongest sampling-based baselines while incurring up to 100x lower computational overhead. Our work demonstrates that LLDMs can deliver both fast inference and reliable hallucination detection simultaneously.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。