用大模型推理拆解复杂图像编辑指令,无需联合训练即可实现精准修改。
Understanding the Implicit User Intention via Reasoning with Large Language Model for Image Editing
- 通过迭代更新机制构建精细视觉表征,分解复杂指令为简单操作
- 在SmartEdit数据集上PSNR提升9.955 dB,显著保留应保持不变区域
- 自建CIEBench基准测试,适合需要推理能力的图像编辑研究者
现有图像编辑方法对简单指令处理良好,但面对复杂指令时通常需联合微调大语言模型(LLM)与扩散模型(DM),计算成本高。为此,我们提出一种新方法CIELR,将复杂用户指令转化为一系列简单明确的编辑动作,避免了联合微调。首先利用基础模型构建输入图像的结构化语义表示,再引入迭代更新机制逐步细化该表示,获得精细的场景视觉表征,从而支持复杂灵活的图像编辑。在SmartEdit Reasoning Scenario Set上的实验表明,本方法在PSNR上比当前最优方法提升9.955 dB,显著提升了需保持一致区域的保真度。由于公开复杂图像编辑带推理的数据集样本有限,我们构建了名为CIEBench的新基准,包含86个图像样本及专门针对推理型编辑的评估指标。CIELR在该基准上同样优于已有方法。代码与数据集已开源。
原文摘要 · Abstract (English)
Existing image editing methods can handle simple editing instructions very well. To deal with complex editing instructions, they often need to jointly fine-tune the large language models (LLMs) and diffusion models (DMs), which involves very high computational complexity and training cost. To address this issue, we propose a new method, called \textbf{C}omplex \textbf{I}mage \textbf{E}diting via \textbf{L}LM \textbf{R}easoning (CIELR), which converts a complex user instruction into a set of simple and explicit editing actions, eliminating the need for jointly fine-tuning the large language models and diffusion models. Specifically, we first construct a structured semantic representation of the input image using foundation models. Then, we introduce an iterative update mechanism that can progressively refine this representation, obtaining a fine-grained visual representation of the image scene. This allows us to perform complex and flexible image editing tasks. Extensive experiments on the SmartEdit Reasoning Scenario Set show that our method surpasses the previous state-of-the-art by 9.955 dB in PSNR, indicating its superior preservation of regions that should remain consistent. Due to the limited number of samples of public datasets of complex image editing with reasoning, we construct a benchmark named CIEBench, containing 86 image samples, together with a metric specifically for reasoning-based image editing. CIELR also outperforms previous methods on this benchmark. The code and dataset are available at \href{https://github.com/Jia-shao/Reasoning-Editing}{https://github.com/Jia-shao/Reasoning-Editing}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。