首次揭示扩散型大模型的幻觉模式与失效机制
Lost in Diffusion: Uncovering Hallucination Patterns and Failure Modes in Diffusion Large Language Models
- 对比分析扩散与自回归模型的幻觉差异
- 发现扩散模型幻觉率更高，且存在早期终止等特有失败模式
- 适合关注生成可靠性与模型安全的研究者
尽管扩散型大语言模型（dLLMs）作为非自回归范式已展现出与自回归（AR）模型相当的潜力，但其生成忠实性，尤其是幻觉问题仍缺乏系统研究。本文首次开展受控对比实验，评估dLLMs中的幻觉模式。结果表明，在架构、规模和预训练权重一致的前提下，当前dLLMs的幻觉倾向显著高于AR模型。进一步分析推理阶段计算资源的使用发现：准自回归生成存在早期饱和，而非序列解码则具备持续优化的潜力。此外，我们识别出扩散过程特有的失效模式，包括过早终止、未完成去噪和上下文干扰。这些发现表明，尽管dLLMs在通用任务上缩小了性能差距，但其独特的幻觉机制对模型可靠性构成重大挑战。代码已开源。
原文摘要 · Abstract (English)
While Diffusion Large Language Models (dLLMs) have emerged as a promising non-autoregressive paradigm comparable to autoregressive (AR) models, their faithfulness, specifically regarding hallucination, remains largely underexplored. To bridge this gap, we present the first controlled comparative study to evaluate hallucination patterns in dLLMs. Our results demonstrate that current dLLMs exhibit a higher propensity for hallucination than AR counterparts controlled for architecture, scale, and pre-training weights. Furthermore, an analysis of inference-time compute reveals divergent dynamics: while quasi-autoregressive generation suffers from early saturation, non-sequential decoding unlocks potential for continuous refinement. Finally, we identify distinct failure modes unique to the diffusion process, including premature termination, incomplete denoising, and context intrusion. Our findings underscore that although dLLMs have narrowed the performance gap on general tasks, their distinct hallucination mechanisms pose a critical challenge to model reliability. Our code is available at https://github.com/ZeroLoss-Lab/Lost-in-Diffusion
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。