提升表征自编码器重建质量,融合语义与低级视觉信息
Improving Reconstruction of Representation Autoencoder
- 用语义特征补充颜色纹理等低级信息,增强重建细节
- 高维隐空间使解码器易受扰动,导致生成伪影
- 通过微调解码器和可控加噪,提升生成鲁棒性
近期工作利用视觉基础模型作为图像编码器,以提升潜在扩散模型(LDMs)的生成性能,因其语义特征分布易于学习。然而,此类语义特征常缺乏低层信息(如颜色、纹理),导致重建保真度下降,成为进一步扩展LDMs的主要瓶颈。为此,我们提出LV-RAE,一种在语义特征基础上补充缺失低层信息的表征自编码器,实现高保真重建的同时保持与语义分布的高度对齐。我们进一步发现,高维且信息丰富的潜在空间使解码器对潜在扰动敏感,导致解码生成潜在表示时出现严重伪影,从而降低生成质量。分析表明,这种敏感性主要源于解码器在数据流形外方向的过度响应。基于此,我们提出微调解码器以提高其鲁棒性,并通过可控噪声注入平滑生成潜在表示,从而提升生成质量。实验表明,LV-RAE显著提升重建保真度,同时保留语义抽象性并实现强生成质量。代码已开源:https://github.com/modyu-liu/LVRAE。
原文摘要 · Abstract (English)
Recent work leverages Vision Foundation Models as image encoders to boost the generative performance of latent diffusion models (LDMs), as their semantic feature distributions are easy to learn. However, such semantic features often lack low-level information (\eg, color and texture), leading to degraded reconstruction fidelity, which has emerged as a primary bottleneck in further scaling LDMs. To address this limitation, we propose LV-RAE, a representation autoencoder that augments semantic features with missing low-level information, enabling high-fidelity reconstruction while remaining highly aligned with the semantic distribution. We further observe that the resulting high-dimensional, information-rich latent make decoders sensitive to latent perturbations, causing severe artifacts when decoding generated latent and consequently degrading generation quality. Our analysis suggests that this sensitivity primarily stems from excessive decoder responses along directions off the data manifold. Building on these insights, we propose fine-tuning the decoder to increase its robustness and smoothing the generated latent via controlled noise injection, thereby enhancing generation quality. Experiments demonstrate that LV-RAE significantly improves reconstruction fidelity while preserving the semantic abstraction and achieving strong generative quality. Our code is available at https://github.com/modyu-liu/LVRAE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。