用改进的统计损失训练生成模型,更好捕捉复杂数据的尾部特征。
Robust training of implicit generative models for multivariate and heavy-tailed distributions with an invariant statistical loss
- 采用广义帕累托噪声与不变统计损失结合,提升对重尾分布的建模能力
- 通过随机投影扩展1维方法,在高维空间中仍能有效建模多变量分布
- 可作为GAN预训练工具,防止模式崩溃,对超参数不敏感,鲁棒性强
传统隐式生成模型依赖对抗判别器区分真实数据与生成数据,常导致训练不稳定和模式丢失。本文基于文献[de2024training]提出的不变统计损失(ISL),将其拓展至处理重尾与多变量数据分布。许多现实世界现象需用重尾概率分布描述,传统隐式方法难以准确刻画其尾部行为。为此,我们提出使用广义帕累托分布(GPD)作为输入噪声的生成器,结合ISL训练,简称Pareto-ISL。实验表明,该方法不仅能精准建模分布尾部,同时保留中心特征。原ISL仅适用于1维数据,直接推广至n维时需处理所有边际分布,计算不可行且在高维下失效。为此,本文引入随机投影,构造适配多变量数据的新损失函数,通过控制投影数量保持问题可解。我们在多维生成建模中评估性能,并探索其作为生成对抗网络(GAN)预训练方法以缓解模式崩溃的潜力,结果表明其在多种超参数设置下均表现稳健,具有较强鲁棒性。
原文摘要 · Abstract (English)
Traditional implicit generative models are capable of learning highly complex data distributions. However, their training involves distinguishing real data from synthetically generated data using adversarial discriminators, which can lead to unstable training dynamics and mode dropping issues. In this work, we build on the \textit{invariant statistical loss} (ISL) method introduced in \cite{de2024training}, and extend it to handle heavy-tailed and multivariate data distributions. The data generated by many real-world phenomena can only be properly characterised using heavy-tailed probability distributions, and traditional implicit methods struggle to effectively capture their asymptotic behavior. To address this problem, we introduce a generator trained with ISL, that uses input noise from a generalised Pareto distribution (GPD). We refer to this generative scheme as Pareto-ISL for conciseness. Our experiments demonstrate that Pareto-ISL accurately models the tails of the distributions while still effectively capturing their central characteristics. The original ISL function was conceived for 1D data sets. When the actual data is $n$-dimensional, a straightforward extension of the method was obtained by targeting the $n$ marginal distributions of the data. This approach is computationally infeasible and ineffective in high-dimensional spaces. To overcome this, we extend the 1D approach using random projections and define a new loss function suited for multivariate data, keeping problems tractable by adjusting the number of projections. We assess its performance in multidimensional generative modeling and explore its potential as a pretraining technique for generative adversarial networks (GANs) to prevent mode collapse, reporting promising results and highlighting its robustness across various hyperparameter settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。