用视觉对应关系指导扩散模型,让虚拟试衣更贴合衣物细节。
Incorporating Visual Correspondence into Diffusion Model for Virtual Try-On
- 将衣物纹理转为语义点,通过局部形变匹配人体对应点。
- 引入3D深度/法线信息增强点对齐,提升细节保真度。
- 适合关注试衣真实感与细节还原的研究者或开发者。
扩散模型在虚拟试衣(VTON)任务中已初见成效,典型双分支架构由两个UNet分别负责隐式服装变形与图像生成。然而,由于扩散模型固有的随机性,保持给定服装的形状和细节仍具挑战。为此,本文提出显式利用视觉对应关系作为先验,以约束扩散过程,而非简单将整件服装输入UNet作为外观参考。具体地,将精细的外观与纹理细节视为一组结构化语义点,通过局部光流形变将衣物上的语义点匹配到目标人体上的对应点。这些2D点进一步结合目标人体的深度图/法线图扩展为3D感知线索。该对应关系模拟了穿衣过程,3D感知线索则作为语义点匹配监督扩散模型训练。此外,设计了面向点的扩散损失以充分挖掘语义点匹配的优势。大量实验表明,本方法在VITON-HD和DressCode数据集上均取得当前最优的试衣性能,显著提升了服装细节保留能力。代码已公开于:https://github.com/HiDream-ai/SPM-Diff。
原文摘要 · Abstract (English)
Diffusion models have shown preliminary success in virtual try-on (VTON) task. The typical dual-branch architecture comprises two UNets for implicit garment deformation and synthesized image generation respectively, and has emerged as the recipe for VTON task. Nevertheless, the problem remains challenging to preserve the shape and every detail of the given garment due to the intrinsic stochasticity of diffusion model. To alleviate this issue, we novelly propose to explicitly capitalize on visual correspondence as the prior to tame diffusion process instead of simply feeding the whole garment into UNet as the appearance reference. Specifically, we interpret the fine-grained appearance and texture details as a set of structured semantic points, and match the semantic points rooted in garment to the ones over target person through local flow warping. Such 2D points are then augmented into 3D-aware cues with depth/normal map of target person. The correspondence mimics the way of putting clothing on human body and the 3D-aware cues act as semantic point matching to supervise diffusion model training. A point-focused diffusion loss is further devised to fully take the advantage of semantic point matching. Extensive experiments demonstrate strong garment detail preservation of our approach, evidenced by state-of-the-art VTON performances on both VITON-HD and DressCode datasets. Code is publicly available at: https://github.com/HiDream-ai/SPM-Diff.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。