重尾噪声虽能提升生成多样性，却会增加采样误差，得不偿失。
Do Heavy Tails Help Diffusion? On the Subtle Trade-off Between Initialization and Training
- 对比重尾与轻尾噪声在扩散模型中的统计估计难度
- 重尾噪声导致采样误差上界更差，理论与实验均验证
- 适合关注生成模型底层机制的研究者阅读
近期研究提出在基于扩散和流的生成模型中引入重尾（HT）噪声，以更好恢复目标分布的尾部并提升生成多样性。这一思路直观合理：若数据本身具有重尾特性，HT噪声可能比轻尾高斯噪声更匹配。然而，用HT噪声替代高斯噪声也改变了底层估计问题。本文通过理论与实证结合的方式重新审视该范式，为两种代表性扩散模型（分别使用HT与轻尾噪声）建立了采样误差边界。结果表明，HT噪声使统计估计问题更难，导致更不利的采样误差上界。我们在合成数据和真实数据集上的实验验证了预测的误差权衡关系。研究质疑了当前生成建模中一种日益增长的设计趋势，挑战了使用HT噪声以探索罕见区域的有效性。
原文摘要 · Abstract (English)
Recent works have proposed incorporating heavy-tailed (HT) noise into diffusion- and flow-based generative models, with the goals of better recovering the tails of target distributions and improving generative diversity. This motivation is intuitive: if the data are heavy-tailed, HT noise may appear better matched than light-tailed (LT) Gaussian noise. However, replacing Gaussian noise by HT noise also changes the underlying estimation problem. In this paper, we revisit this paradigm through a combined theoretical and empirical study, establishing sampling-error bounds for two representative diffusion models driven by HT and LT noise. We show that HT noise makes the statistical estimation problem harder, leading to less favorable sampling-error bounds. We support these findings with experiments on synthetic and real-world datasets, empirically recovering the predicted error trade-off. Our results call into question a growing design trend in generative modeling and challenge the use of HT noise to improve rare-region exploration.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。