通过频谱分析发现自编码器隐空间高频成分干扰生成质量，提出等尺度正则化改进扩散性能。
Improving the Diffusability of Autoencoders
- 提出等尺度正则化，让解码器在不同频率下保持尺度一致性
- 图像生成FID降低19%，视频生成FVD降低44%以上
- 仅需少量微调即可提升扩散模型生成质量，适合部署优化
潜在扩散模型已成为生成高质量图像和视频的主流方法，利用压缩的潜在表示降低扩散过程的计算负担。尽管近期进展主要集中在扩大扩散主干网络和提升自编码器重建质量，但两者之间的相互作用却较少受到关注。本文对现代自编码器进行频谱分析，发现其潜在空间中存在异常的高频成分，尤其在瓶颈通道数较大的自编码器中更为显著。我们假设这些高频成分干扰了扩散合成从粗到细的过程，从而限制生成质量。为缓解此问题，提出等尺度正则化：一种简单有效的正则化策略，通过在解码器中强制尺度等变性，使潜在空间与RGB空间在频率上对齐。该方法仅需少量代码修改和最多20,000次自编码器微调步骤，显著提升生成质量，在ImageNet-1K $256^2$ 上图像生成FID降低19%，在Kinetics-700 $17 \times 256^2$ 上视频生成FVD降低至少44%。
原文摘要 · Abstract (English)
Latent diffusion models have emerged as the leading approach for generating high-quality images and videos, utilizing compressed latent representations to reduce the computational burden of the diffusion process. While recent advancements have primarily focused on scaling diffusion backbones and improving autoencoder reconstruction quality, the interaction between these components has received comparatively less attention. In this work, we perform a spectral analysis of modern autoencoders and identify inordinate high-frequency components in their latent spaces, which are especially pronounced in the autoencoders with a large bottleneck channel size. We hypothesize that this high-frequency component interferes with the coarse-to-fine nature of the diffusion synthesis process and hinders the generation quality. To mitigate the issue, we propose scale equivariance: a simple regularization strategy that aligns latent and RGB spaces across frequencies by enforcing scale equivariance in the decoder. It requires minimal code changes and only up to 20K autoencoder fine-tuning steps, yet significantly improves generation quality, reducing FID by 19% for image generation on ImageNet-1K $256^2$ and FVD by at least 44% for video generation on Kinetics-700 $17 \times 256^2$. The source code is available at https://github.com/snap-research/diffusability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。