首个支持多领域图像修复的统一模型,能自动适配任务与领域需求。
Learning Domain-Aware Task Prompt Representations for Multi-Domain All-in-One Image Restoration
- 通过任务提示池和动态组合机制,自适应生成任务表征。
- 引入领域提示池,融合多模态大模型先验,提升跨域泛化能力。
- 适用于自然、医疗、遥感等多场景修复,适合需要统一处理多种图像问题的用户。
近期,统一图像修复(AiOIR)取得了显著进展,单个模型可处理多种修复任务。然而,现有方法通常局限于特定图像领域,如自然场景、医学成像或遥感。本文首次将AiOIR拓展至多领域,提出DATPRL-IR方法,基于提出的领域感知任务提示表征学习。首先构建包含多个任务提示的任务提示池,隐式编码任务知识;对每张输入图像,模型通过提示组合机制(PCM)自适应选择最相关任务提示,生成实例级任务表征。为进一步赋予模型领域感知能力,引入领域提示池,并从多模态大语言模型中蒸馏领域先验知识注入领域提示。同样使用PCM将自适应选择的领域提示组合为图像级领域表征。最终融合两类表征,形成领域感知任务提示表征,充分利用任务间与领域间的共性与特性知识,指导后续修复过程。大量实验表明,DATPRL-IR显著优于现有最先进方法,具备强泛化能力。代码已开源:https://github.com/GuangluDong0728/DATPRL-IR。
原文摘要 · Abstract (English)
Recently, significant breakthroughs have been made in all-in-one image restoration (AiOIR), which can handle multiple restoration tasks with a single model. However, existing methods typically focus on a specific image domain, such as natural scene, medical imaging, or remote sensing. In this work, we aim to extend AiOIR to multiple domains and propose the first multi-domain all-in-one image restoration method, DATPRL-IR, based on our proposed Domain-Aware Task Prompt Representation Learning. Specifically, we first construct a task prompt pool containing multiple task prompts, in which task-related knowledge is implicitly encoded. For each input image, the model adaptively selects the most relevant task prompts and composes them into an instance-level task representation via a prompt composition mechanism (PCM). Furthermore, to endow the model with domain awareness, we introduce another domain prompt pool and distill domain priors from multimodal large language models into the domain prompts. PCM is utilized to combine the adaptively selected domain prompts into a domain representation for each input image. Finally, the two representations are fused to form a domain-aware task prompt representation which can make full use of both specific and shared knowledge across tasks and domains to guide the subsequent restoration process. Extensive experiments demonstrate that our DATPRL-IR significantly outperforms existing SOTA image restoration methods, while exhibiting strong generalization capabilities. Code is available at https://github.com/GuangluDong0728/DATPRL-IR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。