首次解释变分推断中模式坍缩的成因，并提出旋转改进方法。
Mode Collapse of Mean-Field Variational Inference
- 定义ε-分离度，量化混合成分间距离对坍缩的影响。
- 理论证明当成分过近时，优化器会集中于单一成分，质量占比可低于10%。
- 提出旋转变分推断（RoVI），实测有效缓解模式坍缩问题。
均值场变分推断（MFVI）通过乘积分布近似高维概率分布，但实践中常出现模式坍缩：当目标分布π为混合分布π = wP₀ + (1−w)P₁时，优化器倾向于将大部分质量集中于单一成分。本文首次提供理论解释，引入ε-分离度刻画两成分间分离程度，推导出在ε足够小时，任意MFVI优化器对每个成分的质量分配上限。结果表明，模式坍缩关键取决于成分相对位置。为此，我们提出旋转变分推断（RoVI），在MFVI基础上加入旋转矩阵。数值实验验证了理论发现，并展示了RoVI的有效性。
原文摘要 · Abstract (English)
Mean-field variational inference (MFVI) is a widely used method for approximating high-dimensional probability distributions by product measures. It has been empirically observed that MFVI optimizers often suffer from mode collapse. Specifically, when the target measure $π$ is a mixture $π= w P_0 + (1 - w) P_1$, the MFVI optimizer tends to place most of its mass near a single component of the mixture. This work provides the first theoretical explanation of mode collapse in MFVI. We introduce the notion to capture the separatedness of the two mixture components -- called $\varepsilon$-separateness -- and derive explicit bounds on the fraction of mass that any MFVI optimizer assigns to each component when $P_0$ and $P_1$ are $\varepsilon$-separated for sufficiently small $\varepsilon$. Our results suggest that the occurrence of mode collapse crucially depends on the relative position of the components. To address this issue, we propose the rotational variational inference (RoVI), which augments MFVI with a rotation matrix. The numerical studies support our theoretical findings and demonstrate the benefits of RoVI.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。