无需遮罩即可虚拟试穿任意配饰,突破传统服装限制。
OmniTry: Virtual Try-On Anything without Masks
- 分两阶段训练:先用无配对图像学定位,再用少量配对数据优化外观一致性。
- 仅需少量配对样本即可快速收敛,实测在12类配饰上定位与身份保持更优。
- 适合电商试穿、虚拟穿搭系统开发,支持首饰、配件等复杂物品模拟。
虚拟试穿(VTON)是一项实用且广泛应用的任务,现有方法主要集中于衣物。本文提出OmniTry,一个统一框架,将VTON扩展至任意可穿戴物品,如首饰和配饰,并在无需遮罩的设置下实现更贴近实际应用。当扩展到多种物体类型时,成对图像(物体图与试穿结果图)的获取成为数据构建难点。为此,我们设计了两阶段流程:第一阶段利用大规模无配对图像(即佩戴各类物品的人像)训练模型实现无遮罩定位,具体通过重用图像修复模型,在空掩码中自动定位并绘制物体;第二阶段使用少量成对图像微调模型,以传递物体外观的一致性。实验发现,第一阶段后的模型即使在少量配对样本下也能快速收敛。OmniTry在包含12类常见可穿戴物品的综合基准上评估,涵盖商店内与野外图像。结果表明,其在物体定位和身份保持方面均优于现有方法。代码、模型权重及评测基准将公开发布于https://omnitry.github.io/。
原文摘要 · Abstract (English)
Virtual Try-ON (VTON) is a practical and widely-applied task, for which most of existing works focus on clothes. This paper presents OmniTry, a unified framework that extends VTON beyond garment to encompass any wearable objects, e.g., jewelries and accessories, with mask-free setting for more practical application. When extending to various types of objects, data curation is challenging for obtaining paired images, i.e., the object image and the corresponding try-on result. To tackle this problem, we propose a two-staged pipeline: For the first stage, we leverage large-scale unpaired images, i.e., portraits with any wearable items, to train the model for mask-free localization. Specifically, we repurpose the inpainting model to automatically draw objects in suitable positions given an empty mask. For the second stage, the model is further fine-tuned with paired images to transfer the consistency of object appearance. We observed that the model after the first stage shows quick convergence even with few paired samples. OmniTry is evaluated on a comprehensive benchmark consisting of 12 common classes of wearable objects, with both in-shop and in-the-wild images. Experimental results suggest that OmniTry shows better performance on both object localization and ID-preservation compared with existing methods. The code, model weights, and evaluation benchmark of OmniTry will be made publicly available at https://omnitry.github.io/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。