arXiv:2508.03587cs.LG2025-08被引 1

通过特定解码器架构实现零方差梯度,稳定VAE训练

Zero-Variance Gradients for Variational Autoencoders

  • 设计特殊解码器结构,使证据下界可解析计算
  • 相比标准方法,收敛更快且生成质量更优
  • 适合追求训练稳定性与高效性的生成模型研究者

训练深度生成模型如变分自编码器(VAEs)需通过随机潜在变量传播梯度,这会引入估计方差,影响收敛速度与性能。本文提出一种新思路——静默梯度(Silent Gradients)。通过限制解码器架构,使期望的证据下界(ELBO)可解析计算,从而获得零估计方差的梯度,无需依赖潜在变量的蒙特卡洛采样。我们首先在具有线性解码器的受控设置中进行理论分析,证明其优化效果优于标准估计器。为推广至高表达能力的非线性解码器,提出一种训练范式:先用解析梯度引导早期编码器学习,再逐步退火至标准随机估计器。在多个数据集上,该方法持续优于重参数化、Gumbel-Softmax和REINFORCE等基线方法。结果表明,支持解析期望计算的架构选择能显著稳定含随机组件的生成模型训练。

原文摘要 · Abstract (English)

Training deep generative models like Variational Autoencoders (VAEs) requires propagating gradients through stochastic latent variables, which introduces estimation variance that can slow convergence and degrade performance. In this paper, we explore an orthogonal direction, which we call Silent Gradients. Instead of designing improved stochastic estimators, we show that by restricting the decoder architecture in specific ways, the expected ELBO can be computed analytically. This yields gradients with zero estimation variance as we can directly compute the evidence lower-bound without resorting to Monte Carlo samples of the latent variables. We first provide a theoretical analysis in a controlled setting with a linear decoder and demonstrate improved optimization compared to standard estimators. To extend this idea to expressive nonlinear decoders, we introduce a training paradigm that uses the analytic gradient to guide early encoder learning before annealing to a standard stochastic estimator. Across multiple datasets, our approach consistently improves established baselines, including reparameterization, Gumbel-Softmax, and REINFORCE. These results suggest that architectural choices enabling analytic expectation computation can significantly stabilize the training of generative models with stochastic components.

变分自编码器零方差梯度生成模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。