针对文生图模型中的人体伪影问题，提出新数据集与检测模型。
Detecting Human Artifacts from Text-to-Image Models
- 构建3.7万张图像的专用数据集HAD，标注人体伪影位置。
- 训练出跨生成模型通用的伪影检测模型，准确识别各类人体异常。
- 可用于优化生成模型和修复任意图像中的伪影，适合视觉质量研究者。
尽管近期进展显著，文生图模型生成的人像常出现伪影，如肢体扭曲、缺失或多余部分，严重破坏人体结构一致性并降低图像保真度。为此，我们构建了人类伪影数据集（HAD），包含超过37,000张由多个主流文生图模型生成的图像，并标注了人体伪影位置。基于此数据集，我们训练了人类伪影检测模型（HADM），可跨多种生成域识别不同类型的伪影，且在未见生成器上仍具强泛化能力。进一步地，我们利用HADM预测结果作为反馈，对扩散模型进行微调，实验表明生成图像中的人体伪影显著减少。此外，我们展示了一种新颖应用：在迭代修复框架中直接修正任意图像中的人体伪影，验证其提升图像质量的能力。相关数据集与模型已开源：https://github.com/wangkaihong/HADM。
原文摘要 · Abstract (English)
Despite recent advancements, text-to-image generation models often produce images containing artifacts, especially in human figures. These artifacts appear as poorly generated human bodies, including distorted, missing, or extra body parts, leading to visual inconsistencies with typical human anatomy and greatly impairing overall fidelity. In this study, we address this challenge by curating Human Artifact Dataset (HAD), a diverse dataset specifically designed to localize human artifacts. HAD comprises over 37,000 images generated by several popular text-to-image models, annotated for human artifact localization. Using this dataset, we train the Human Artifact Detection Models (HADM), which can identify different artifacts across multiple generative domains and demonstrate strong generalization, even on images from unseen generators. Additionally, to further improve generators' perception of human structural coherence, we use the predictions from our HADM as feedback for diffusion model finetuning. Our experiments confirm a reduction in human artifacts in the resulting model. Furthermore, we showcase a novel application of our HADM in an iterative inpainting framework to correct human artifacts in arbitrary images directly, demonstrating its utility in improving image quality. Our dataset and detection models are available at: https://github.com/wangkaihong/HADM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。