提出生成先于重建的训练策略,实现稳定端到端潜空间生成建模。
GenFirst: Generation Before Reconstruction for Stable End-to-End Latent Generative Modeling
- 先用生成目标塑造潜空间,再逐步增强重建约束。
- 在ImageNet-256上达gFID 0.97(带CFG)和1.45(无CFG)。
- 适用于图像、图文联合生成等多模态任务,通用性强。
潜空间生成模型通常采用两阶段流程:先训练变分自编码器进行重构,再在冻结的潜空间上训练生成模型。由于重构优化的潜变量未必适合生成,联合训练两者更具吸引力。然而,直接端到端训练仍具挑战性,易引发潜变量坍塌,并存在生成与重构的冲突。本文通过分析不同目标对潜空间的影响,发现两个关键洞察:首先,KL散度中的熵项对防止坍塌至关重要;重建与先验拟合会压缩后验分布,而熵项维持潜变量的非退化不确定性。其次,生成与重构的学习动态不对称:重构快速且强监督,而生成更慢且难优化。基于此,我们首次实现无潜变量坍塌的直接端到端训练,提出GenFirst——一种简单的生成优先重建策略。生成目标先在弱重构压力下塑造潜空间,随后逐步强化重构以恢复视觉细节。我们在具有精确似然的连续自回归先验和具有隐式似然的SiT先验上验证了GenFirst。结合端到端目标与GenFirst,SiT在ImageNet-256上达到带CFG的gFID 0.97、不带CFG的1.45;MMDiT在文本到图像生成上取得GenEval 0.90。该框架还可扩展至共享视觉潜空间的生成与表征学习,以及连续统一文本-图像生成。结果表明,稳定端到端潜空间学习在多种生成先验与模态中均具普适性。
原文摘要 · Abstract (English)
Latent generative models typically follow a two-stage pipeline, training a variational autoencoder for reconstruction and then a generative model on the frozen latent space. Since reconstruction-optimized latents are not necessarily generation-friendly, jointly training both models is an appealing alternative. However, direct end-to-end training remains challenging, as it is prone to latent collapse and faces a generation-reconstruction conflict. We revisit this problem by analyzing how different objectives shape the latent space and identify two key insights. First, the entropy term in the Kullback-Leibler divergence objective is essential for preventing collapse: reconstruction and prior fitting tend to shrink the posterior, while entropy preserves non-degenerate latent uncertainty. Second, reconstruction and generation exhibit asymmetric learning dynamics: reconstruction is fast and strongly supervised, whereas generation is slower and harder to optimize. Based on these insights, we achieve the first direct end-to-end training without latent collapse and propose GenFirst, a simple generation-before-reconstruction strategy. The generative objective first shapes the latent space under weak reconstruction pressure, after which reconstruction is progressively strengthened to recover visual details. We validate GenFirst with continuous autoregressive priors with exact likelihoods and SiT priors with implicit likelihoods. With our end-to-end objective and GenFirst, SiT achieves a gFID of 0.97 with CFG and 1.45 without CFG on ImageNet-256, while MMDiT reaches a GenEval score of 0.90 on text-to-image generation. Beyond image generation, we extend the framework to shared visual latents for generation and representation learning, and to continuous unified text-image generation. These results demonstrate the generality of stable end-to-end latent learning across generative priors and modalities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。