用图文交替推理提升图像编辑精度,解决复杂布局与细节生成难题。
Beyond Textual CoT: Interleaved Text-Image Chains with Deep Confidence Reasoning for Image Editing
- 采用图文交替的思维链,分步结合文本描述与视觉线索进行编辑。
- 在三个基准上显著优于现有方法,最高提升18.3%(PSNR)。
- 适合需要精细控制图像内容与空间关系的研究者与开发者。
自然语言驱动的图像编辑日益流行,但现有方法在处理复杂对象交集和细粒度空间关系时表现不佳,源于缺乏显式推理过程。尽管已探索使用思维链(CoT)增强推理能力,但纯文本CoT或仅附加坐标信息的方法在表征复杂视觉布局方面存在根本局限,且缺乏引导像素级细节生成所需的视觉线索。为此,我们提出多模态推理编辑(MURE)框架,将编辑过程从纯文本推理转向一系列交错的文本-视觉推理链。该框架通过原生多模态的图文交替思维链实现图像编辑,在每一步先输出文本描述,再提供对应视觉提示(如目标编辑区域的位置掩码或新内容表示)。为缓解大模型幻觉问题,引入多模态深度置信度(MMDC)推理范式:在每一步探索视觉推理路径树,并利用奖励模型生成的深度置信分数剪枝低质量分支,确保模型始终沿高质量轨迹推进至最终结果。该方法将复杂编辑任务分解为相互依赖的子任务,提升各阶段精度,实现高保真编辑效果。我们定义了图文交错链的形式化表达,并发布首个CoT-Edit-14K数据集,包含14,000个高质量编辑样本。大量实验表明,该方法在三个图像编辑基准上均取得显著提升。
原文摘要 · Abstract (English)
Image editing with natural language has gained significant popularity, yet existing methods struggle with intricate object intersections and fine-grained spatial relationships due to the lack of an explicit reasoning process. While Chain-of-Thought (CoT) has been explored to enhance reasoning, purely textual CoT or CoT augmented with coordinate information is fundamentally limited in its ability to represent intricate visual layouts and lacks the necessary visual cues to guide the generation of fine-grained, pixel-level details. To address these challenges, we propose Multimodal Reasoning Edit (MURE), a novel framework that shifts the visual editing process from purely text-based reasoning to a series of interleaved textual and visual rationales. Our framework performs image editing using a natively multimodal, interleaved text-image CoT. This approach generates a step-by-step chain of reasoning where a textual description is followed by a corresponding visual cue, such as a positional mask that defined intended edited regions or a representation of new content. Furthermore, to mitigate the hallucination phenomenon of large language models, we introduce Multimodal Deep Confidence (MMDC) reasoning paradigm. This paradigm explores a tree of visual reasoning paths at each step. By pruning low-quality branches using a deep confidence score from a reward model, it ensures the model consistently follows a high-quality trajectory towards the final edited result. The proposed method decomposes complex editing tasks into interdependent sub-tasks, achieving greater precision at each stage and yielding high-fidelity edited results. We define the formulation for interleaved text-image chains and release the first CoT-Edit-14K dataset, comprising 14K high-quality editing examples. Extensive experiments show that our method yields significant improvements across three image editing benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。