用自然梯度改进混合密度网络训练，提速十倍且不增加开销
Learning Mixture Density via Natural Gradient Expectation Maximization
- 将混合密度网络视为隐变量模型，用期望最大化框架分析优化路径
- 提出nGEM方法，实测收敛速度提升最高达10倍，高维数据仍有效
- 适合需要快速训练多模态概率模型的研究者，尤其处理高维数据
混合密度网络是生成连续多模态条件分布的神经网络，通常通过负对数似然（NLL）进行最大似然估计，但存在收敛慢和模式崩溃问题。本文从信息几何角度改进其优化：将混合密度网络视为深层隐变量模型，基于期望最大化框架分析，发现其与自然梯度下降存在意外理论联系。据此推导出自然梯度期望最大化（nGEM）目标函数。实验表明，nGEM在几乎无额外计算开销下实现最高10倍的收敛加速，并可有效扩展至高维数据，而传统NLL在此类场景下失效。
原文摘要 · Abstract (English)
Mixture density networks are neural networks that produce Gaussian mixtures to represent continuous multimodal conditional densities. Standard training procedures involve maximum likelihood estimation using the negative log-likelihood (NLL) objective, which suffers from slow convergence and mode collapse. In this work, we improve the optimization of mixture density networks by integrating their information geometry. Specifically, we interpret mixture density networks as deep latent-variable models and analyze them through an expectation maximization framework, which reveals surprising theoretical connections to natural gradient descent. We then exploit such connections to derive the natural gradient expectation maximization (nGEM) objective. We show that empirically nGEM achieves up to 10$\times$ faster convergence while adding almost zerocomputational overhead, and scales well to high-dimensional data where NLL otherwise fails.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。