arXiv:2410.18666cs.CV2024-10NeurIPS被引 73

用百万级隐私安全数据+自适应扩散模型,实现真实世界图像修复

DreamClear: High-Capacity Real-World Image Restoration with Privacy-Safe Dataset Curation

  • 构建百万张高质量图像数据集,通过双提示学习自动生成并过滤内容
  • 提出基于扩散变换器的DreamClear模型,在多种退化下实现逼真修复
  • 适合需要高精度图像修复的研究者与工业应用开发者

真实世界图像修复面临模型容量不足和数据集有限的挑战。为此,我们提出双策略:GenIR 数据清洗流水线与基于扩散变换器(DiT)的DreamClear图像修复模型。GenIR 采用三阶段双提示学习流程——图像-文本对构建、基于双提示的微调、数据生成与筛选,避免人工爬取,保障版权与隐私,生成一百万张高质量图像数据。DreamClear 利用文生图扩散模型的生成先验与多模态大语言模型的感知能力,实现逼真修复,并引入可适应调制混合模块(MoAM),通过逐标记退化先验动态融合不同修复专家,提升对多样化真实退化场景的适应性。大量实验验证了该方法在真实世界图像修复中的优越性能。代码与预训练模型已开源。

原文摘要 · Abstract (English)

Image restoration (IR) in real-world scenarios presents significant challenges due to the lack of high-capacity models and comprehensive datasets. To tackle these issues, we present a dual strategy: GenIR, an innovative data curation pipeline, and DreamClear, a cutting-edge Diffusion Transformer (DiT)-based image restoration model. GenIR, our pioneering contribution, is a dual-prompt learning pipeline that overcomes the limitations of existing datasets, which typically comprise only a few thousand images and thus offer limited generalizability for larger models. GenIR streamlines the process into three stages: image-text pair construction, dual-prompt based fine-tuning, and data generation & filtering. This approach circumvents the laborious data crawling process, ensuring copyright compliance and providing a cost-effective, privacy-safe solution for IR dataset construction. The result is a large-scale dataset of one million high-quality images. Our second contribution, DreamClear, is a DiT-based image restoration model. It utilizes the generative priors of text-to-image (T2I) diffusion models and the robust perceptual capabilities of multi-modal large language models (MLLMs) to achieve photorealistic restoration. To boost the model's adaptability to diverse real-world degradations, we introduce the Mixture of Adaptive Modulator (MoAM). It employs token-wise degradation priors to dynamically integrate various restoration experts, thereby expanding the range of degradations the model can address. Our exhaustive experiments confirm DreamClear's superior performance, underlining the efficacy of our dual strategy for real-world image restoration. Code and pre-trained models are available at: https://github.com/shallowdream204/DreamClear.

图像修复扩散模型数据构建隐私安全

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。