用文本描述精准修复图像缺失区域，解决错位与风格不一致问题。
MTADiffusion: Mask Text Alignment Diffusion Model for Object Inpainting
- 通过自动标注生成500万图文对，提升修复语义准确性。
- 引入边缘预测和风格一致性损失，修复结果更结构稳定、风格统一。
- 在两个基准测试中表现领先，适合需要高精度修复的场景。
生成模型的发展使得图像修复能根据提示和掩码生成特定区域内容。但现有方法常存在语义错位、结构失真和风格不一致问题。本文提出MTADiffusion，一种基于掩码-文本对齐的扩散模型用于物体修复。为增强语义能力，我们设计MTAPipeline，自动为掩码添加详细描述，并构建包含500万图像和2500万掩码-文本对的新数据集MTADataset。提出多任务训练策略，融合修复与边缘预测任务以提升结构稳定性；并引入基于预训练VGG和格拉姆矩阵的风格一致性损失，改善风格一致性。在BrushBench和EditBench上的全面评估显示，MTADiffusion性能优于现有方法。
原文摘要 · Abstract (English)
Advancements in generative models have enabled image inpainting models to generate content within specific regions of an image based on provided prompts and masks. However, existing inpainting methods often suffer from problems such as semantic misalignment, structural distortion, and style inconsistency. In this work, we present MTADiffusion, a Mask-Text Alignment diffusion model designed for object inpainting. To enhance the semantic capabilities of the inpainting model, we introduce MTAPipeline, an automatic solution for annotating masks with detailed descriptions. Based on the MTAPipeline, we construct a new MTADataset comprising 5 million images and 25 million mask-text pairs. Furthermore, we propose a multi-task training strategy that integrates both inpainting and edge prediction tasks to improve structural stability. To promote style consistency, we present a novel inpainting style-consistency loss using a pre-trained VGG network and the Gram matrix. Comprehensive evaluations on BrushBench and EditBench demonstrate that MTADiffusion achieves state-of-the-art performance compared to other methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。