通过理论分析发现，合理设计退火策略可有效防止变分推断中的模式崩溃。
Annealing in variational inference mitigates mode collapse: A theoretical study on Gaussian mixtures
- 基于高斯混合模型，用低维统计量刻画退火温度与速率的交互作用。
- 推导出模式崩溃概率的精确公式，证明恰当退火能稳定避免模式丢失。
- 结果对神经网络和归一化流模型具指导意义，适合研究变分推断的从业者。
模式崩溃是现代变分推断中难以捕捉多模态分布某一或多个模式的核心挑战。本文在可解析的设定下——学习高斯混合模型（Gaussian Mixture）——对基于退火的策略进行数学分析，该设定中模式崩溃现象已知存在。借助低维总结统计量描述，我们精确刻画了初始温度与退火速率之间的相互作用，并推导出模式崩溃概率的精确公式。分析表明，合理设计的退火方案可稳健地防止模式崩溃。最后，我们提供了数值证据，表明这些理论权衡在基于神经网络的模型和RealNVP归一化流中具有定性推广性，为实际变分推断流程中缓解模式崩溃的退火策略设计提供了指导。
原文摘要 · Abstract (English)
Mode collapse, the failure to capture one or more modes when targetting a multimodal distribution, is a central challenge in modern variational inference. In this work, we provide a mathematical analysis of annealing based strategies for mitigating mode collapse in a tractable setting: learning a Gaussian mixture, where mode collapse is known to arise. Leveraging a low dimensional summary statistics description, we precisely characterize the interplay between the initial temperature and the annealing rate, and derive a sharp formula for the probability of mode collapse. Our analysis shows that an appropriately chosen annealing scheme can robustly prevent mode collapse. Finally, we present numerical evidence that these theoretical tradeoffs qualitatively extend to neural network based models, RealNVP normalizing flows, providing guidance for designing annealing strategies mitigating mode collapse in practical variational inference pipelines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。