用神经网络学习凸输运势函数,实现精准概率分布生成。
Learning Brenier Potentials with Convex Generative Adversarial Neural Networks
- 设计带立方激活的ReCU网络,兼具霍尔德函数逼近与密度平滑特性。
- 引入凸性惩罚项的对抗训练,确保生成网络始终严格凸。
- 理论证明渐进收敛,实验覆盖高斯混合到图像数据,效果稳定。
Brenier证明,在源分布与目标分布满足一定条件时,存在一个严格凸函数,其梯度可作为从源分布到目标分布的输运映射,该函数称为Brenier势函数。本文建立了学习此类势函数的生成对抗神经网络的统计学习理论。由于生成分布的密度依赖于势函数的二阶导数,我们提出了具有立方激活函数 ReCU(x)=max{0,x}^3 的神经网络,其兼具霍尔德函数的良好逼近能力与密度的Lipschitz连续性。为保证网络的凸性,设计了一种对抗训练方法:在经典判别器交叉熵损失基础上加入凸性惩罚项。通过学习误差的详细分解,证明当惩罚参数足够大时,所有参与对抗极小极大优化的网络均严格凸。进而证明了网络容量缓慢增长下的学习过程一致性。我们实现了该算法,并应用于标准测试案例(从高斯混合到图像数据)。理论预测结果成立:训练过程中凸性惩罚项趋于失效,而网络所表示的势函数已成功学习到凸性。
原文摘要 · Abstract (English)
Brenier proved that under certain conditions on a source and a target probability measure there exists a strictly convex function such that its gradient is a transport map from the source to the target distribution. This function is called the Brenier potential. Furthermore, detailed information on the Hölder regularity of the Brenier potential is available. In this work we develop the statistical learning theory of generative adversarial neural networks that learn the Brenier potential. As by the transformation of densities formula, the density of the generated measure depends on the second derivative of the Brenier potential, we develop the universal approximation theory of ReCU networks with cubic activation $\mathtt{ReCU}(x)=\max\{0,x\}^3$ that combines the favorable approximation properties of Hölder functions with a Lipschitz continuous density. In order to assure the convexity of such general networks, we introduce an adversarial training procedure for a potential function represented by the ReCU networks that combines the classical discriminator cross entropy loss with a penalty term that enforces (strict) convexity. We give a detailed decomposition of learning errors and show that for a suitable high penalty parameter all networks chosen in the adversarial min-max optimization problem are strictly convex. This is further exploited to prove the consistency of the learning procedure for (slowly) expanding network capacity. We also implement the described learning algorithm and apply it to a number of standard test cases from Gaussian mixture to image data as target distributions. As predicted in theory, we observe that the convexity loss becomes inactive during the training process and the potentials represented by the neural networks have learned convexity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。