用服装细节增强扩散模型,让虚拟试穿更真实。
Improving Virtual Try-On with Garment-focused Diffusion Models
- 基于参考服装的外观和纹理,强化扩散过程中的局部细节生成
- 在VITON-HD和DressCode数据集上超越现有方法
- 适合需要高保真服装细节生成的研究与应用
扩散模型在图像生成任务中取得了革命性进展,但直接用于图像驱动的虚拟试穿(VTON)仍具挑战:需同时生成目标人物的高质量逼真图像,并精确保留给定服装的每一处外观与纹理细节。为此,本文提出新模型GarDiff,通过放大参考服装的视觉外观与高频率纹理信息的引导,实现以服装为中心的扩散过程。GarDiff首先利用参考服装的CLIP与VAE编码引入额外外观先验,对预训练潜空间扩散模型进行重制;同时,在扩散模型的UNet中集成新型服装聚焦适配器,实现与参考服装视觉特征及人体姿态的细粒度局部对齐。特别设计了针对合成服装的外观损失函数,以增强关键的高频细节。在VITON-HD与DressCode数据集上的大量实验表明,GarDiff显著优于当前最先进的VTON方法。代码已公开于https://github.com/siqi0905/GarDiff/tree/master。
原文摘要 · Abstract (English)
Diffusion models have led to the revolutionizing of generative modeling in numerous image synthesis tasks. Nevertheless, it is not trivial to directly apply diffusion models for synthesizing an image of a target person wearing a given in-shop garment, i.e., image-based virtual try-on (VTON) task. The difficulty originates from the aspect that the diffusion process should not only produce holistically high-fidelity photorealistic image of the target person, but also locally preserve every appearance and texture detail of the given garment. To address this, we shape a new Diffusion model, namely GarDiff, which triggers the garment-focused diffusion process with amplified guidance of both basic visual appearance and detailed textures (i.e., high-frequency details) derived from the given garment. GarDiff first remoulds a pre-trained latent diffusion model with additional appearance priors derived from the CLIP and VAE encodings of the reference garment. Meanwhile, a novel garment-focused adapter is integrated into the UNet of diffusion model, pursuing local fine-grained alignment with the visual appearance of reference garment and human pose. We specifically design an appearance loss over the synthesized garment to enhance the crucial, high-frequency details. Extensive experiments on VITON-HD and DressCode datasets demonstrate the superiority of our GarDiff when compared to state-of-the-art VTON approaches. Code is publicly available at: \href{https://github.com/siqi0905/GarDiff/tree/master}{https://github.com/siqi0905/GarDiff/tree/master}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。