发现扩散语言模型评估存在幻觉,提示模板影响结果可靠性。
Understanding Evaluation Illusion in Diffusion Large Language Models

- 通过多模板测试揭示解码方法排名受提示模板影响极大。
- 现有并行解码均不如单步解码,无法突破速度与质量权衡。
- 提出评估指南,避免因设置不当导致错误结论。
尽管具备并行解码能力,扩散大语言模型(dLLMs)仍需大量去噪步骤以保证生成质量,促使研究高效解码策略。然而,现有研究在看似相同的评估条件下却报告不一致结果,可能引发对解码方法的偏误判断。为此,我们在多种评估设置下对当前dLLM解码方法进行严谨评测。令人意外的是,解码方法的排序高度依赖于提示模板选择。单一模板评估可能导致虚假结论,即认为解码方法提升了推理效率而未损失性能。全面实验表明,当前并行解码方法始终弱于单标记解码基线,未能克服速度-质量权衡。我们进一步发现,这种评估不一致性源于并行解码对提示模板微小变化的高度敏感性。实验显示,一个有效提示模板可在较少去噪步数下实现优异表现,显著优于增加去噪步数带来的边际收益。此外,被忽视的评估设置也显著影响方法评估结果。基于此,我们提出适用于dLLM解码方法的可靠评估实践指南。
原文摘要 · Abstract (English)
Despite the capability of parallel decoding, diffusion large language models (dLLMs) require many denoising steps to maintain generation quality, motivating recent research on efficient decoding strategies. However, existing studies have reported inconsistent evaluation results even under seemingly identical evaluation settings, risking biased conclusions about dLLM decoding methods. To understand this evaluation concern, we conduct a rigorous evaluation of current decoding methods for dLLMs across diverse evaluation settings. Surprisingly, our analysis reveals that the ranking of decoding methods is highly sensitive to the choice of prompt templates. Single-template evaluation can lead to an illusion that decoding methods improve inference efficiency without performance degradation. Through comprehensive experiments, we find that current parallel decoding methods consistently underperform the single-token decoding baseline, failing to overcome the speed-quality trade-off. We further identify this evaluation inconsistency as the high sensitivity of parallel decoding methods to minor variations in prompt templates. Our experiments show that an effective prompt template can achieve strong evaluation results even with fewer denoising steps, markedly outperforming the marginal gain from increasing denoising steps. Beyond prompt templates, our experiments indicate that overlooked evaluation settings can also notably affect the assessment of decoding methods. Based on these findings, we propose practical guidelines for the reliable evaluation of decoding methods in dLLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。