直接从原始传感器数据恢复高清图像,效果更好更少瑕疵。
RDDM: Practicing RAW Domain Diffusion Model for Real-world Image Restoration
- 在原始数据域直接生成图像,跳过传统处理流程。
- 相比现有方法,图像保真度更高,伪影更少。
- 适合边缘设备图像恢复,支持多种传感器格式。
我们提出一种原始数据域扩散模型(RDDM),可直接从传感器原始数据中端到端恢复逼真图像。尽管现有sRGB域扩散方法表现优异,但其处理有损的sRGB输入,且忽略了边缘设备中原始数据的可用性,导致性能受限。RDDM通过在原始域直接恢复图像,摆脱了传统的两阶段图像信号处理(ISP)->图像恢复(IR)流程。为解决预训练扩散模型直接适配原始域带来的挑战,我们提出:(1) 原始域变分自编码器(RVAE),将传感器原始数据编码并解码为增强的线性域图像,缓解不同域分布间的分布外(OOD)问题;(2) 可配置多拜耳(CMB)LoRA模块,支持RGGB、BGGR等多种原始拜耳格式。为弥补数据不足,我们构建了可扩展的数据合成流水线,基于现有sRGB数据集生成大规模原始低质-高质对用于训练。大量实验表明,RDDM在保真度和伪影控制上均优于当前最先进的sRGB扩散方法。代码将在 https://github.com/YanCHEN-fr/RDDM 公开。
原文摘要 · Abstract (English)
We present the RAW domain diffusion model (RDDM), an end-to-end diffusion model that restores photo-realistic images directly from the sensor RAW data. While recent sRGB-domain diffusion methods achieve impressive results, they are caught in a dilemma between high fidelity and image generation. These models process lossy sRGB inputs and neglect the accessibility of the sensor RAW images in many scenarios, e.g., in image and video capturing in edge devices, resulting in sub-optimal performance. RDDM obviates this limitation by directly restoring images in the RAW domain, replacing the conventional two-stage image signal processing (ISP)->Image Restoration (IR) pipeline. However, a simple adaptation of pre-trained diffusion models to the RAW domain confronts many challenges. To this end, we propose: (1) a RAW-domain VAE (RVAE), encoding sensor RAW and decoding it into an enhanced linear domain image, to solve the out-of-distribution (OOD) issues between the different domain distributions; (2) a configurable multi-bayer (CMB) LoRA module, adapting diverse RAW Bayer patterns such as RGGB, BGGR, etc. To compensate for the deficiency in the dataset, we develop a scalable data synthesis pipeline synthesizing RAW LQ-HQ pairs from existing sRGB datasets for large-scale training. Extensive experiments demonstrate RDDM's superiority over state-of-the-art sRGB diffusion methods, yielding higher fidelity results with fewer artifacts. Codes will be publicly available at https://github.com/YanCHEN-fr/RDDM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。