用大模型精准控制图像物体编辑,减少人工操作。
POEM: Precise Object-level Editing via MLLM control
- 利用多模态大模型解析指令,自动生成编辑前后的物体掩码。
- 在PASCAL VOC基础上构建新数据集,验证了编辑精度显著提升。
- 适合需要精细图像修改且不想手动标注的用户使用。
扩散模型在文本到图像生成方面取得显著进展,能够根据文字描述生成高质量、逼真的图像。然而,在生成之外,物体级图像编辑仍具挑战性,需在保持视觉连贯性的前提下实现精确修改。现有基于文本的编辑方法在局部形状和布局变换上表现不佳,常引发不必要的全局变化。基于图像交互的方法虽精度更高,但需大量人工输入以提供精确指导。为减少人工成本同时保持高编辑精度,本文提出POEM框架,即基于多模态大语言模型(MLLM)的精确物体级编辑方法。POEM利用MLLM分析指令提示,生成编辑前后物体的精确掩码,通过结构化推理阶段引导扩散模型编辑过程,确保物体准确定位与变换。为评估该方法,我们构建了基于PASCAL VOC 2012的基准数据集VOCEdits,包含指令编辑提示、真实变换标签和精确物体掩码。实验结果表明,相较于现有文本驱动方法,POEM在精度与可靠性上均有提升;相比交互式方法,其人工投入更少。
原文摘要 · Abstract (English)
Diffusion models have significantly improved text-to-image generation, producing high-quality, realistic images from textual descriptions. Beyond generation, object-level image editing remains a challenging problem, requiring precise modifications while preserving visual coherence. Existing text-based instructional editing methods struggle with localized shape and layout transformations, often introducing unintended global changes. Image interaction-based approaches offer better accuracy but require manual human effort to provide precise guidance. To reduce this manual effort while maintaining a high image editing accuracy, in this paper, we propose POEM, a framework for Precise Object-level Editing using Multimodal Large Language Models (MLLMs). POEM leverages MLLMs to analyze instructional prompts and generate precise object masks before and after transformation, enabling fine-grained control without extensive user input. This structured reasoning stage guides the diffusion-based editing process, ensuring accurate object localization and transformation. To evaluate our approach, we introduce VOCEdits, a benchmark dataset based on PASCAL VOC 2012, augmented with instructional edit prompts, ground-truth transformations, and precise object masks. Experimental results show that POEM outperforms existing text-based image editing approaches in precision and reliability while reducing manual effort compared to interaction-based methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。