让扩散模型生成更真实:用原始像素信息纠正细节失真
When Latents Forget Pixels: Restoring Fidelity in Diffusion Transformer Super-Resolution

- 用上采样前的原始像素补全压缩丢失的细节
- 双阶段注入像素线索,提升生成图像与输入的一致性
- 轻量改造大模型,适合想提升真实感的视觉研究者
大型生成模型在图像超分辨率任务中虽取得优异感知质量,但保持与低分辨率输入的一致性仍具挑战。我们发现基于潜在表示的扩散变换器(DiTs)存在关键缺陷:VAE压缩瓶颈削弱了细粒度空间信息,导致生成细节与输入关联弱。为此,本文提出像素锚定超分辨率(PGSR)框架,在VAE压缩前保留低分辨率图像的像素证据,并在重建过程中分两阶段复用。首先,条件侧轨迹引导将像素证据与潜变量条件融合,指导潜在空间修复路径;其次,解码侧像素锚定将多尺度像素特征注入冻结的VAE解码器,强化最终渲染的输入依据。为高效适配预训练大模型,仅训练轻量级修复模块,冻结主干网络与潜在自编码器,并引入局部窗口注意力提升高分辨率效率。大量实验表明,PGSR显著改善真实感与保真度权衡,生成结果更具真实性与说服力。
原文摘要 · Abstract (English)
Image super-resolution (SR) with large generative models has recently achieved remarkable perceptual quality, yet maintaining fidelity to the LR observation remains challenging. In particular, we observe that diffusion transformers (DiTs) built on latent representations suffer from a critical limitation: the compression bottleneck of the VAE weakens fine-grained spatial information, leading to hallucinated details that are weakly grounded in the input image. In this work, we revisit generative SR from a representation perspective and propose a pixel-grounded super-resolution (PGSR) framework that preserves LR-observed pixel evidence before VAE compression and reuses it throughout restoration. Instead of relying solely on the compressed latent condition, PGSR extracts pre-VAE pixel evidence from the upsampled LR image and reuses it at two stages. First, Condition-Side Trajectory Guidance fuses LR-derived pixel evidence with the latent LR condition to guide the latent restoration trajectory. Second, Decoder-Side Pixel Grounding injects multi-scale pixel features into the frozen VAE decoder to ground the final rendering with LR-observed cues. To efficiently adapt large pretrained DiT models, we keep the latent autoencoder and main flow-matching backbone frozen, and train only lightweight restoration modules. We further study an efficient local-window attention variant for improved high-resolution efficiency and scalability. Extensive experiments demonstrate that PGSR improves the realism--fidelity trade-off and produces more faithful, visually convincing results than existing latent generative SR approaches.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。