提出新方法实现隐式生成模型的显式密度近似,训练更稳定且防模式崩溃。
Explicit Density Approximation for Neural Implicit Samplers Using a Bernstein-Based Convex Divergence
- 通过交换目标与模型分布角色,构建凸优化的新型无似然损失函数。
- 理论证明该方法收敛更快、训练更平稳,且能避免模式崩溃。
- 首次提供密度近似的闭式表达,适合需精确密度建模的研究者。
基于排名的统计度量(如不变统计损失ISL)已成为训练隐式生成模型的有效工具。本文提出双侧ISL(dual-ISL),在ISL框架中互换目标分布与模型分布的角色,使模型密度空间中的优化问题变为凸问题。我们证明所得的秩相关差异 $d_K$ 具备:一是在弱收敛和 $L^1$ 范数下连续;二是对第一变量凸性——这些性质是经典散度(如KL或Wasserstein距离)所不具备的。在此基础上,我们建立理论框架,将 $d_K$ 视为密度比 $q = p/ ilde p$ 在Bernstein多项式基下的 $L^2$-投影,推导出截断误差的精确界、明确的收敛速率,并获得截断密度近似的闭式表达。进一步通过随机一维投影扩展至多变量情形,定义了保持凸性和连续性的切片双侧ISL散度。实验表明,这些理论优势转化为实际性能提升:在多个基准测试中,dual-ISL收敛更快、训练更平滑稳定,有效防止模式崩溃,且优于经典ISL及其他主流隐式生成方法,同时提供显式密度近似。
原文摘要 · Abstract (English)
Rank-based statistical metrics, such as the invariant statistical loss (ISL), have recently emerged as robust and practically effective tools for training implicit generative models. In this work, we introduce dual-ISL, a novel likelihood-free objective for training implicit generative models that interchanges the roles of the target and model distributions in the ISL framework, yielding a convex optimization problem in the space of model densities. We prove that the resulting rank-based discrepancy $d_K$ is i) continuous under weak convergence and with respect to the $L^1$ norm, and ii) convex in its first argument-properties not shared by classical divergences such as KL or Wasserstein distances. Building on this, we develop a theoretical framework that interprets $d_K$ as an $L^2$-projection of the density ratio $q = p/\tilde p$ onto a Bernstein polynomial basis, from which we derive exact bounds on the truncation error, precise convergence rates, and a closed-form expression for the truncated density approximation. We further extend our analysis to the multivariate setting via random one-dimensional projections, defining a sliced dual-ISL divergence that retains both convexity and continuity. We empirically show that these theoretical advantages translate into practical ones. Specifically, across several benchmarks dual-ISL converges more rapidly, delivers markedly smoother and more stable training, and more effectively prevents mode collapse than classical ISL and other leading implicit generative methods-while also providing an explicit density approximation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。