修复扩散模型生成的手部畸形，保持原图风格与姿态
HandCraft: Anatomically Correct Restoration of Malformed Hands in Diffusion Generated Images
- 用参数化模型自动生成手部掩码和深度图作为条件信号
- 无需微调即可修复手部解剖结构与姿势，还原自然形态
- 适用于各类扩散模型，适合需要高质量手部图像的场景
生成式文本到图像模型（如 Stable Diffusion）虽能生成多样且高质量的图像，但在表现人手时却常出现解剖结构错误或进入“恐怖谷”效应。本文提出 HandCraft 方法，通过参数化模型自动构建手部掩码和深度图作为条件信号，驱动基于扩散的图像编辑器修复手部解剖结构并调整姿态，同时无缝融合到原图中，保持原始姿态、颜色与风格。该方案为即插即用式，无需对预训练扩散模型进行微调或训练。我们还构建了 MalHand 数据集，包含多种风格下存在各种手部畸形的生成图像，用于手部检测器训练与修复效果评估。定性与定量实验表明，HandCraft 不仅恢复手部解剖正确性，还保持整体图像完整性。
原文摘要 · Abstract (English)
Generative text-to-image models, such as Stable Diffusion, have demonstrated a remarkable ability to generate diverse, high-quality images. However, they are surprisingly inept when it comes to rendering human hands, which are often anatomically incorrect or reside in the "uncanny valley". In this paper, we propose a method HandCraft for restoring such malformed hands. This is achieved by automatically constructing masks and depth images for hands as conditioning signals using a parametric model, allowing a diffusion-based image editor to fix the hand's anatomy and adjust its pose while seamlessly integrating the changes into the original image, preserving pose, color, and style. Our plug-and-play hand restoration solution is compatible with existing pretrained diffusion models, and the restoration process facilitates adoption by eschewing any fine-tuning or training requirements for the diffusion models. We also contribute MalHand datasets that contain generated images with a wide variety of malformed hands in several styles for hand detector training and hand restoration benchmarking, and demonstrate through qualitative and quantitative evaluation that HandCraft not only restores anatomical correctness but also maintains the integrity of the overall image.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。