通过频谱残差修正提升扩散模型生成质量，轻量高效且无需额外训练
SpectralDiT: Timestep-Conditioned Spectral Residual Correction for Flow-Matching DiTs
- 在残差分支中引入时步条件的频谱分解修正机制
- CIFAR-10上FID从20.78降至19.71，图像频谱更均匀
- 适用于高保真图像生成，尤其适合资源受限场景
我们提出SpectralDiT，一种针对流匹配扩散Transformer的轻量级改进方法，通过在MLP残差分支中添加时步条件的频谱校正模块，将每个残差更新在补丁-令牌网格上分解为低频与高频分量，并学习一个零初始化的加性门控机制，使模型初始阶段保持基线性能。在像素空间生成任务中，SpectralDiT在CIFAR-10上将FID从20.78降至19.71（补丁大小为1），并缩小了径向傅里叶谱差距。进一步扩展至ImageNet-100的潜在扩散模型，仅增加0.6%理论浮点运算量和1.36%参数量，便在分类器自由引导（CFG 2.0）下实现8.7%的相对FID下降。所有结果均基于五次随机种子平均。消融实验与门控可视化揭示了稳定且块特定的频谱修正模式。
原文摘要 · Abstract (English)
We propose SpectralDiT, a lightweight modification to flow-matching Diffusion Transformers that adds timestep-conditioned spectral correction to the MLP residual branch. The module decomposes each residual update into low- and high-frequency components on the patch-token grid, then learns a zero-initialized additive gate so the model initially matches the baseline DiT. On CIFAR-10 pixel-space generation, SpectralDiT improves FID from 20.78 to 19.71 at patch size 1 and reduces the radial Fourier spectrum gap. Furthermore, we scale our method to latent diffusion on ImageNet-100. With 0.6% additional theoretical FLOPs and 1.36% additional parameters, SpectralDiT improves latent flow-matching, achieving an 8.7% relative FID reduction under classifier-free guidance (CFG 2.0). All reported results are averaged over five seeds. Ablations and gate visualizations on CIFAR-10 reveal stable block-specific spectral correction patterns.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。