arXiv:2503.01103cs.CVcs.LG2025-03ICML被引 22

用反向KL和自生成负样本提升生成模型质量,突破最大似然训练瓶颈。

Direct Discriminative Optimization: Your Likelihood-Based Visual Generative Model is Secretly a GAN Discriminator

  • 通过隐式参数化判别器,将似然训练与GAN思想结合,无需联合训练。
  • 在多个数据集上将扩散模型FID降低至新纪录,降幅达27%以上。
  • 适合追求高质量生成的科研人员和工程师,尤其擅长无引导生成任务。

尽管基于似然的生成模型(如扩散模型和自回归模型)在视觉生成中取得了显著的保真度,但最大似然估计(MLE)目标最小化前向KL散度,固有地具有模式覆盖倾向,在模型容量有限时会限制生成质量。本文提出直接判别优化(DDO),作为一种统一框架,将似然生成训练与GAN式判别相结合,通过利用反向KL散度和自生成负信号来绕过这一根本限制。核心思想是使用可学习目标模型与固定参考模型之间的似然比隐式参数化判别器,借鉴直接偏好优化(DPO)的哲学。与GAN不同,该参数化无需联合训练生成器与判别器网络,可直接高效地对已训练模型进行微调,使其性能超越MLE极限。DDO可通过自对弈方式迭代进行,每轮仅需少于1%的预训练周期。实验表明,DDO显著提升了现有SOTA扩散模型EDM,在CIFAR-10/ImageNet-64/ImageNet 512x512上的FID分别从1.79/1.58/1.96降至1.30/0.97/1.26,且无需任何引导机制;同时一致提升了ImageNet 256x256上自回归模型的无引导和CFG增强版FID。

原文摘要 · Abstract (English)

While likelihood-based generative models, particularly diffusion and autoregressive models, have achieved remarkable fidelity in visual generation, the maximum likelihood estimation (MLE) objective, which minimizes the forward KL divergence, inherently suffers from a mode-covering tendency that limits the generation quality under limited model capacity. In this work, we propose Direct Discriminative Optimization (DDO) as a unified framework that integrates likelihood-based generative training and GAN-type discrimination to bypass this fundamental constraint by exploiting reverse KL and self-generated negative signals. Our key insight is to parameterize a discriminator implicitly using the likelihood ratio between a learnable target model and a fixed reference model, drawing parallels with the philosophy of Direct Preference Optimization (DPO). Unlike GANs, this parameterization eliminates the need for joint training of generator and discriminator networks, allowing for direct, efficient, and effective finetuning of a well-trained model to its full potential beyond the limits of MLE. DDO can be performed iteratively in a self-play manner for progressive model refinement, with each round requiring less than 1% of pretraining epochs. Our experiments demonstrate the effectiveness of DDO by significantly advancing the previous SOTA diffusion model EDM, reducing FID scores from 1.79/1.58/1.96 to new records of 1.30/0.97/1.26 on CIFAR-10/ImageNet-64/ImageNet 512x512 datasets without any guidance mechanisms, and by consistently improving both guidance-free and CFG-enhanced FIDs of visual autoregressive models on ImageNet 256x256.

扩散模型生成模型优化方法

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。