用底层特征替代文本,提升真实照片超分辨率效果。
F2IDiff: Real-world Image Super-resolution using Feature to Image Diffusion Foundation Model
- 用DINOv2特征代替文本作为扩散模型条件输入
- 在12MP以上手机照片上实现更精准无幻觉重建
- 适合追求真实细节的消费级图像增强场景
生成式AI使单图超分辨率(SISR)质量显著提升,因文本到图像扩散基础模型(T2IDiff FM)所学强先验可弥合高分辨(HR)与低分辨(LR)图像差距。但旗舰手机相机迟迟未采用生成模型,因强生成易引发不可接受的幻觉。对于学术界常见严重退化的LR图像,强生成虽有幻觉但仍可接受,因LR与HR差距大;而在消费摄影中,LR图像保真度高,仅需极少无幻觉生成即可。我们假设SISR中的生成受基础模型条件特征的严格性与丰富性控制。首先,文本特征为高层特征,难以描述图像细微纹理;其次,智能手机LR图像至少12MP,而基于T2IDiff FM构建的SISR网络设计用于处理远小于1MP的图像,因此必须在小块上推理,难以由文本特征准确描述。为解决此问题,我们引入一种基于下层特征条件的SISR网络,使用DINOv2特征,称为特征到图像扩散(F2IDiff)基础模型(FM)。低层级特征提供更严格约束,同时能丰富描述小块图像。
原文摘要 · Abstract (English)
With the advent of Generative AI, Single Image Super-Resolution (SISR) quality has seen substantial improvement, as the strong priors learned by Text-2-Image Diffusion (T2IDiff) Foundation Models (FM) can bridge the gap between High-Resolution (HR) and Low-Resolution (LR) images. However, flagship smartphone cameras have been slow to adopt generative models because strong generation can lead to undesirable hallucinations. For substantially degraded LR images, as seen in academia, strong generation is required and hallucinations are more tolerable because of the wide gap between LR and HR images. In contrast, in consumer photography, the LR image has substantially higher fidelity, requiring only minimal hallucination-free generation. We hypothesize that generation in SISR is controlled by the stringency and richness of the FM's conditioning feature. First, text features are high level features, which often cannot describe subtle textures in an image. Additionally, Smartphone LR images are at least $12MP$, whereas SISR networks built on T2IDiff FM are designed to perform inference on much smaller images ($<1MP$). As a result, SISR inference has to be performed on small patches, which often cannot be accurately described by text feature. To address these shortcomings, we introduce an SISR network built on a FM with lower-level feature conditioning, specifically DINOv2 features, which we call a Feature-to-Image Diffusion (F2IDiff) Foundation Model (FM). Lower level features provide stricter conditioning while being rich descriptors of even small patches.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。