通过反向对齐提升生成模型的语义表示与生成质量。
Flowing Backwards: Improving Normalizing Flows via Reverse Representation Alignment
- 用逆向生成过程的中间特征对齐视觉基础模型表示。
- 训练速度提升3.3倍,图像生成与分类性能均达新纪录。
- 无需训练的测试时优化法,可直接评估模型语义能力。
归一化流(Normalizing Flows, NFs)是一类具有数学可逆结构的生成模型,前向传播将数据映射到隐空间用于密度估计,反向传播则从该空间生成新样本。这一特性使表征学习与数据生成天然协同。然而,标准NFs因对数似然优化导致语义表征不佳,限制了生成质量。为此,我们提出一种创新对齐策略:不正则化前向过程,而是将生成(反向)过程的中间特征与强大视觉基础模型的表示对齐,效果显著优于简单对齐。此外,我们引入一种无需训练的测试时优化算法,用于分类任务,可更内在地评估NF嵌入的语义知识。大量实验表明,该方法使NF训练加速超过3.3倍,同时在生成质量与分类准确率上均有显著提升。在ImageNet 64×64和256×256上均达到当前最佳性能。代码已公开于https://github.com/MCG-NJU/FlowBack。
原文摘要 · Abstract (English)
Normalizing Flows (NFs) are a class of generative models distinguished by a mathematically invertible architecture, where the forward pass transforms data into a latent space for density estimation, and the reverse pass generates new samples from this space. This characteristic creates an intrinsic synergy between representation learning and data generation. However, the generative quality of standard NFs is limited by poor semantic representations from log-likelihood optimization. To remedy this, we propose a novel alignment strategy that creatively leverages the invertibility of NFs: instead of regularizing the forward pass, we align the intermediate features of the generative (reverse) pass with representations from a powerful vision foundation model, demonstrating superior effectiveness over naive alignment. We also introduce a novel training-free, test-time optimization algorithm for classification, which provides a more intrinsic evaluation of the NF's embedded semantic knowledge. Comprehensive experiments demonstrate that our approach accelerates the training of NFs by over 3.3$\times$, while simultaneously delivering significant improvements in both generative quality and classification accuracy. New state-of-the-art results for NFs are established on ImageNet 64$\times$64 and 256$\times$256. Our code is available at https://github.com/MCG-NJU/FlowBack.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。