提出λ-VAE解决变分自编码器后验崩溃问题
$\mathbfλ$-VAE: Variance Equalization for Posterior Collapse

- 通过调整采样噪声缩放实现潜变量方差均衡
- 实验显示潜变量信息量最高提升2.8倍,重建误差降低0.33 BPD
- 适合关注生成模型稳定训练的研究者
变分自编码器(VAEs)常遭遇后验崩溃问题,即近似后验趋近先验,导致潜在代码无信息量。我们识别并形式化了两个逻辑独立但耦合的原因:梯度失衡指解码器重构信号衰减速率快于KL正则压力;信息间隙指随机采样过程丢弃编码器计算的大量表示,削弱解码器敏感性,使崩溃代价低廉。两者共享相同崩溃轨迹,且信息间隙在代数上等价于聚合后验与先验的不匹配,统一了两种病理。随后提出λ-VAE,仅修改重参数化步骤:采样噪声按维度指数缩放,而KL惩罚保留原始后验方差。这种不对称性将稳定训练吸引子移出退化崩溃状态,推动所有潜变量维度趋向同一平衡——称为方差均衡。基于净信息增益目标推导出每维度闭式最优指数,仅需一个超参数控制重构与生成权衡。在标准基准(二值MNIST、二值Omniglot、CIFAR-10、CelebA-64)上验证,持续减少崩溃维度,信息容量最高提升2.8×纳特,重建质量最高改善0.33 BPD。
原文摘要 · Abstract (English)
Variational Autoencoders (VAEs) frequently suffer from posterior collapse, a failure mode in which the approximate posterior converges to the prior, rendering the latent code uninformative. Despite extensive research, a unified account of why collapse occurs has remained an open question. We identify and formalize two logically independent but coupled causes. \emph{Gradient imbalance} occurs when the decoder's reconstruction signal vanishes faster than the $\mathbb{KL}$ regularization pressure as the posterior widens. \emph{Information gap} occurs when the stochastic sampling step discards a substantial fraction of the encoder's computed representation, attenuating decoder sensitivity and making collapse inexpensive. Both causes share the same collapse trajectory, and we show that the information gap is algebraically equivalent to mismatch between the aggregate posterior and the prior, unifying two pathologies. Subsequently, we introduce $λ$-VAE, which resolves both causes through a single modification to the reparameterization step: the sampling noise is scaled by per-dimension exponent, while the $\mathbb{KL}$ penalty retains the original posterior variance. This asymmetry shifts the stable training attractor away from the degenerate collapsed state, driving all latent dimensions toward the same equilibrium -- a mechanism we term \emph{variance equalization}. A closed-form optimal exponent per dimension follows from a net information gain objective, with a single hyperparameter controlling the reconstruction-generation tradeoff. We validate on standard benchmarks (Binary MNIST, Binary Omniglot, CIFAR-10, CelebA-64), showing consistent reductions in collapsed dimensions, information capacity gains of up to $2.8\times$ nats, and reconstruction quality improvements of up to $+0.33$ BPD.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。