用视频修复思路解决试穿一致性难题,效果更自然。
Video Virtual Try-on with Conditional Diffusion Transformer Inpainter
- 将试穿转为条件视频修复任务,从源头提升时空一致性。
- 3D时空注意力+多阶段训练,在多个数据集上超越现有方法。
- 适合需要高一致性的视频试穿应用,如虚拟购物、影视特效。
视频虚拟试穿旨在自然地将服装适配到连续视频帧中的目标人物。该任务极具挑战性:一方面要求输出视频具有良好的时空一致性,另一方面还需在所有帧中保留服装细节。直接逐帧使用图像级试穿方法会导致严重不一致。尽管少数基于扩散模型的视频试穿方法尝试引入时间注意力以适应视频任务,仍存在一致性问题。本文提出ViTI(Video Try-on Inpainter),将视频虚拟试穿建模为条件视频修复任务,与以往方法不同。通过构建基于扩散变换器的视频修复框架,采用全3D时空注意力机制,并结合多种掩码策略与多阶段训练,逐步适应服装修复任务。模型可依据提示,在掩码区域中填充符合预期的服装像素,保持良好时空一致性。最后引入服装条件控制,确保生成服装外观和细节符合要求。定量与定性实验均表明,ViTI优于现有方法。
原文摘要 · Abstract (English)
Video virtual try-on aims to naturally fit a garment to a target person in consecutive video frames. It is a challenging task, on the one hand, the output video should be in good spatial-temporal consistency, on the other hand, the details of the given garment need to be preserved well in all the frames. Naively using image-based try-on methods frame by frame can get poor results due to severe inconsistency. Recent diffusion-based video try-on methods, though very few, happen to coincide with a similar solution: inserting temporal attention into image-based try-on model to adapt it for video try-on task, which have shown improvements but there still exist inconsistency problems. In this paper, we propose ViTI (Video Try-on Inpainter), formulate and implement video virtual try-on as a conditional video inpainting task, which is different from previous methods. In this way, we start with a video generation problem instead of an image-based try-on problem, which from the beginning has a better spatial-temporal consistency. Specifically, at first we build a video inpainting framework based on Diffusion Transformer with full 3D spatial-temporal attention, and then we progressively adapt it for video garment inpainting, with a collection of masking strategies and multi-stage training. After these steps, the model can inpaint the masked garment area with appropriate garment pixels according to the prompt with good spatial-temporal consistency. Finally, as other try-on methods, garment condition is added to the model to make sure the inpainted garment appearance and details are as expected. Both quantitative and qualitative experimental results show that ViTI is superior to previous works.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。