让用户用指令定制图像修复区域,实现精准局部增强。
InstructRestore: Region-Customized Image Restoration with Human Instructions
- 基于人类指令生成区域描述与掩码,构建53.7万组训练数据。
- 在ControlNet框架下动态调节不同区域的细节增强程度。
- 支持如虚化效果保留、局部强化等交互式修复需求。
尽管基于扩散先验的图像修复技术取得显著进展,但现有方法通常对整幅图像采用统一处理,无法根据用户指令实现区域定制化修复。本文提出InstructRestore框架,实现遵循人类指令的区域可调图像修复。为此,我们设计数据生成引擎,创建包含高质量图像、目标区域描述和对应掩码的三元组,经严格筛选后构建包含536,945组样本的综合性数据集,支持该任务的训练与评估。进一步研究如何在ControlNet架构下融合低质量图像特征,以调节图像细节增强程度。据此开发类似ControlNet的模型,识别目标区域并为不同区域分配不同的融合尺度,实现与用户指令一致的区域定制化修复。实验表明,InstructRestore能有效实现人机交互式修复,例如保留背景虚化效果或按指令进行局部增强。本工作推动了交互式图像修复与增强技术的发展。数据、代码与模型见:https://github.com/shuaizhengliu/InstructRestore.git。
原文摘要 · Abstract (English)
Despite the significant progress in diffusion prior-based image restoration, most existing methods apply uniform processing to the entire image, lacking the capability to perform region-customized image restoration according to user instructions. In this work, we propose a new framework, namely InstructRestore, to perform region-adjustable image restoration following human instructions. To achieve this, we first develop a data generation engine to produce training triplets, each consisting of a high-quality image, the target region description, and the corresponding region mask. With this engine and careful data screening, we construct a comprehensive dataset comprising 536,945 triplets to support the training and evaluation of this task. We then examine how to integrate the low-quality image features under the ControlNet architecture to adjust the degree of image details enhancement. Consequently, we develop a ControlNet-like model to identify the target region and allocate different integration scales to the target and surrounding regions, enabling region-customized image restoration that aligns with user instructions. Experimental results demonstrate that our proposed InstructRestore approach enables effective human-instructed image restoration, such as images with bokeh effects and user-instructed local enhancement. Our work advances the investigation of interactive image restoration and enhancement techniques. Data, code, and models will be found at https://github.com/shuaizhengliu/InstructRestore.git.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。