arXiv:2603.14645cs.CV2026-03被引 5

通过频谱匹配提升潜在空间扩散性能,让生成图像更真实自然。

Spectrum Matching: a Unified Perspective for Superior Diffusability in Latent Diffusion

  • 提出频谱匹配新思路:编码与解码阶段分别对齐图像与潜在表示的功率谱密度。
  • 在CelebA和ImageNet上生成质量优于现有方法,有效缓解过噪或过平滑问题。
  • 适用于追求高质量图像生成的开发者,尤其适合扩散模型优化场景。

本文研究变分自编码器(VAE)在潜在扩散中的可扩散性(学习能力)。首先发现,以MSE为损失的像素空间扩散固有偏向于学习低频和中频成分,而自然图像的幂律功率谱密度使这种偏差在感知上具有优势。基于此,我们提出“频谱匹配假说”:具有良好可扩散性的潜在表示应满足(i)遵循平坦化的幂律功率谱密度(编码频谱匹配,ESM),及(ii)通过解码器保持频率到频率的语义对应关系(解码频谱匹配,DSM)。实践中,通过图像与潜在表示间功率谱密度匹配实现ESM,利用频率对齐的共享频谱掩码进行重建来实现DSM。频谱匹配统一解释了先前关于过噪或过平滑潜在表示的观察,并将多个近期方法(如VA-VAE、EQ-VAE)视为特例。实验表明,该方法在CelebA和ImageNet数据集上均取得更优的扩散生成效果。最后,我们将频谱视角扩展至表征对齐(REPA):揭示目标表征的方向性频谱能量至关重要,并提出基于DoG的方法进一步提升性能。代码已开源。

原文摘要 · Abstract (English)

In this paper, we study the diffusability (learnability) of variational autoencoders (VAE) in latent diffusion. First, we show that pixel-space diffusion trained with an MSE objective is inherently biased toward learning low and mid spatial frequencies, and that the power-law power spectral density (PSD) of natural images makes this bias perceptually beneficial. Motivated by this result, we propose the \emph{Spectrum Matching Hypothesis}: latents with superior diffusability should (i) follow a flattened power-law PSD (\emph{Encoding Spectrum Matching}, ESM) and (ii) preserve frequency-to-frequency semantic correspondence through the decoder (\emph{Decoding Spectrum Matching}, DSM). In practice, we apply ESM by matching the PSD between images and latents, and DSM via shared spectral masking with frequency-aligned reconstruction. Importantly, Spectrum Matching provides a unified view that clarifies prior observations of over-noisy or over-smoothed latents, and interprets several recent methods as special cases (e.g., VA-VAE, EQ-VAE). Experiments suggest that Spectrum Matching yields superior diffusion generation on CelebA and ImageNet datasets, and outperforms prior approaches. Finally, we extend the spectral view to representation alignment (REPA): we show that the directional spectral energy of the target representation is crucial for REPA, and propose a DoG-based method to further improve the performance of REPA. Our code is available https://github.com/forever208/SpectrumMatching.

扩散模型频谱匹配图像生成潜在空间

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。