用语义对齐与不确定性引导自动优化AI图像修复质量。
SAGI: Semantically Aligned and Uncertainty Guided AI Image Inpainting
- 通过预训练模型采样贴近人类感知的提示词，提升生成一致性。
- 使人类识别修复图与真实图的准确率从74%降至35%。
- 适用于提升图像伪造检测性能，对抗生成式AI滥用。
生成式AI的发展使文本引导的图像修复（通过文本提示添加、删除或修改图像区域）变得普及。然而，生成语义正确且逼真的图像通常需要精心设计的提示词，并通过人工反复评估生成内容的真实性——这一过程费时费力。为此，我们提出一种模型无关的管道SAGI（语义对齐与不确定性引导的AI图像修复），从贴近人类感知的分布中采样提示词，并利用预训练的大语言模型和视觉-语言模型近似该分布，评估生成内容并剔除偏离者。在多个顶尖修复模型上应用此管道，构建了目前最大最多样化的AI生成修复数据集SAGI-D，包含超过9.5万张修复图像及人工评估子集。实验表明，语义对齐显著提升图像质量和美感，不确定性引导有效识别真实操作：人类区分修复图与真实图的准确率从74%下降至35%。此外，使用SAGI-D训练多个图像取证方法，在域内检测平均提升37.4%的交并比（IoU），域外泛化能力提升26.1%，证明其在防范生成式AI滥用方面的价值。代码与数据集见https://mever-team.github.io/SAGI/
原文摘要 · Abstract (English)
Recent advancements in generative AI have made text-guided image inpainting - adding, removing, or altering image regions using textual prompts - widely accessible. However, generating semantically correct photorealistic imagery, typically requires carefully-crafted prompts and iterative refinement by evaluating the realism of the generated content - tasks commonly performed by humans. To automate the generative process, we propose Semantically Aligned and Uncertainty Guided AI Image Inpainting (SAGI), a model-agnostic pipeline, to sample prompts from a distribution that closely aligns with human perception and to evaluate the generated content and discard instances that deviate from such a distribution, which we approximate using pretrained large language models and vision-language models. By applying this pipeline on multiple state-of-the-art inpainting models, we create the SAGI Dataset (SAGI-D), currently the largest and most diverse dataset of AI-generated inpaintings, comprising over 95k inpainted images and a human-evaluated subset. Our experiments show that semantic alignment significantly improves image quality and aesthetics, while uncertainty guidance effectively identifies realistic manipulations - human ability to distinguish inpainted images from real ones drops from 74% to 35% in terms of accuracy, after applying our pipeline. Moreover, using SAGI-D for training several image forensic approaches increases in-domain detection performance on average by 37.4% and out-of-domain generalization by 26.1% in terms of IoU, also demonstrating its utility in countering malicious exploitation of generative AI. Code and dataset are available at https://mever-team.github.io/SAGI/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。