用大模型驱动智能代理,实现精准反复的图像编辑。
An LLM-LVLM Driven Agent for Iterative and Fine-Grained Image Editing
- 用LLM拆解任务、LVLM理解指令与场景,闭环迭代优化
- 在新基准上达3.67分,显著优于现有方法
- 适合需要精细修改和持续调整的视觉创作场景
尽管文本到图像生成模型能力强大,但现实应用常需细粒度、迭代式图像编辑,现有方法难以满足。关键挑战包括精准理解指令、修改时保持上下文一致,以及缺乏智能反馈机制。本文提出RefineEdit-Agent,一种无需训练的智能代理框架,通过大语言模型(LLM)的规划能力和视觉语言大模型(LVLM)的视觉理解与评估能力,在闭环系统中实现复杂、迭代、上下文感知的图像编辑。框架包含LVLM驱动的指令解析与场景理解模块、多层级LLM驱动的编辑规划模块(目标分解、工具选择、序列生成)、迭代编辑模块,以及关键的LVLM驱动反馈评估环。为严格评估,我们构建LongBench-T2I-Edit基准,包含500张初始图像,涵盖九种视觉维度的复杂多轮编辑指令。实验表明,RefineEdit-Agent平均得分3.67,显著优于基线:直接重提示(2.29)、InstructPix2Pix(2.91)、基于GLIGEN的编辑(3.16)和ControlNet-XL(3.39)。消融研究、人工评估及对迭代优化、主干模型、工具使用、指令复杂度鲁棒性的分析,进一步验证了该代理设计在提升编辑保真度与上下文一致性方面的有效性。
原文摘要 · Abstract (English)
Despite the remarkable capabilities of text-to-image (T2I) generation models, real-world applications often demand fine-grained, iterative image editing that existing methods struggle to provide. Key challenges include granular instruction understanding, robust context preservation during modifications, and the lack of intelligent feedback mechanisms for iterative refinement. This paper introduces RefineEdit-Agent, a novel, training-free intelligent agent framework designed to address these limitations by enabling complex, iterative, and context-aware image editing. RefineEdit-Agent leverages the powerful planning capabilities of Large Language Models (LLMs) and the advanced visual understanding and evaluation prowess of Vision-Language Large Models (LVLMs) within a closed-loop system. Our framework comprises an LVLM-driven instruction parser and scene understanding module, a multi-level LLM-driven editing planner for goal decomposition, tool selection, and sequence generation, an iterative image editing module, and a crucial LVLM-driven feedback and evaluation loop. To rigorously evaluate RefineEdit-Agent, we propose LongBench-T2I-Edit, a new benchmark featuring 500 initial images with complex, multi-turn editing instructions across nine visual dimensions. Extensive experiments demonstrate that RefineEdit-Agent significantly outperforms state-of-the-art baselines, achieving an average score of 3.67 on LongBench-T2I-Edit, compared to 2.29 for Direct Re-Prompting, 2.91 for InstructPix2Pix, 3.16 for GLIGEN-based Edit, and 3.39 for ControlNet-XL. Ablation studies, human evaluations, and analyses of iterative refinement, backbone choices, tool usage, and robustness to instruction complexity further validate the efficacy of our agentic design in delivering superior edit fidelity and context preservation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。