用扩散变压器实现极简虚拟试穿,效果好还快。
MC-VTON: Minimal Control Virtual Try-On Diffusion Transformer
- 用扩散变压器直接处理人像和服装图,无需额外网络
- 仅需8步推理,参数增量仅0.72%,细节还原更真实
- 适合追求高效、低资源的虚拟试穿应用开发
基于扩散模型的虚拟试穿方法虽能生成逼真效果,但需额外参考网络或图像编码器处理多条件输入,增加预处理复杂度与计算开销,且通常需要超过25步推理,耗时较长。本文提出MC-VTON,借助扩散变压器(DiT)的内在结构,无缝集成极简条件输入。相比现有方法,优势体现在四方面:(1) 更优的细节保真度;(2) 网络与输入简化,无需长提示、姿态估计、人体分割、深度图等冗余条件,仅需遮挡人像与服装图;(3) 参数高效训练,微调FLUX.1-dev仅增加3970万参数(占骨干参数0.33%);(4) 推理步骤减少,通过蒸馏扩散仅需8步即可生成真实感试穿图,额外参数8680万(占骨干参数0.72%)。实验表明,MC-VTON在条件输入、可训练参数和推理步数均更少的情况下,仍取得更优定性与定量结果。
原文摘要 · Abstract (English)
Virtual try-on methods based on diffusion models achieve realistic try-on effects. They use an extra reference network or an additional image encoder to process multiple conditional image inputs, which adds complexity pre-processing and additional computational costs. Besides, they require more than 25 inference steps, bringing longer inference time. In this work, with the development of diffusion transformer (DiT), we rethink the necessity of additional reference network or image encoder and introduce MC-VTON, which leverages DiT's intrinsic backbone to seamlessly integrate minimal conditional try-on inputs. Compared to existing methods, the superiority of MC-VTON is demonstrated in four aspects: (1) Superior detail fidelity. Our DiT-based MC-VTON exhibits superior fidelity in preserving fine-grained details. (2) Simplified network and inputs. We remove any extra reference network or image encoder. We also remove unnecessary conditions like the long prompt, pose estimation, human parsing, and depth map. We require only the masked person image and the garment image. (3) Parameter-efficient training. To process the try-on task, we fine-tune the FLUX.1-dev with only 39.7M additional parameters (0.33% of the backbone parameters). (4) Less inference steps. We apply distillation diffusion on MC-VTON and only need 8 steps to generate a realistic try-on image, with only 86.8M additional parameters (0.72% of the backbone parameters). Experiments show that MC-VTON achieves superior qualitative and quantitative results with fewer condition inputs, trainable parameters, and inference steps than baseline methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。