用多提示生成让画作修复更智能，支持多个区域同时修复。
I Dream My Painting: Connecting MLLMs and Diffusion Models via Prompt Generation for Text-Guided Multi-Mask Inpainting
- 用多模态大模型自动为多个缺失区域生成详细提示。
- 在WikiArt和DCI数据集上实现高精度、创意性修复结果。
- 适合艺术修复与图像编辑领域研究者使用。
图像修复旨在填补图像中缺失或损坏区域，使其与周围内容和风格自然融合。尽管条件扩散模型在文本引导修复中表现优异，本文提出全新任务——多掩码修复，即通过不同提示同时修复多个区域。我们设计了针对多模态大模型（如LLaVA）的微调方法，利用带损图像作为输入，自动生成适用于各掩码区域的详细提示。这些提示随后输入经过修正交叉注意力微调的Stable Diffusion模型，确保提示精准作用于对应区域。在WikiArt和Densely Captioned Images数据集上的实验表明，该流程能生成富有创意且准确的修复结果。代码、数据及训练模型已公开于https://cilabuniba.github.io/i-dream-my-painting。
原文摘要 · Abstract (English)
Inpainting focuses on filling missing or corrupted regions of an image to blend seamlessly with its surrounding content and style. While conditional diffusion models have proven effective for text-guided inpainting, we introduce the novel task of multi-mask inpainting, where multiple regions are simultaneously inpainted using distinct prompts. Furthermore, we design a fine-tuning procedure for multimodal LLMs, such as LLaVA, to generate multi-mask prompts automatically using corrupted images as inputs. These models can generate helpful and detailed prompt suggestions for filling the masked regions. The generated prompts are then fed to Stable Diffusion, which is fine-tuned for the multi-mask inpainting problem using rectified cross-attention, enforcing prompts onto their designated regions for filling. Experiments on digitized paintings from WikiArt and the Densely Captioned Images dataset demonstrate that our pipeline delivers creative and accurate inpainting results. Our code, data, and trained models are available at https://cilabuniba.github.io/i-dream-my-painting.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。