arXiv:2412.10178cs.CVcs.AI2024-12

用扩散模型实现快速一致的视频虚拟试穿,解决画面抖动和计算冗余问题。

SwiftTry: Fast and Consistent Video Virtual Try-On with Diffusion Models

  • 将视频试穿转为条件视频修复,引入时序注意力提升连贯性。
  • 新方法在长视频上推理速度提升3倍,帧间一致性显著改善。
  • 适用于电商直播、服装设计等需要高效生成的场景。

给定一段人物视频和一件新衣物,本文目标是合成一段该人物穿上指定衣物的视频,同时保持时空一致性。尽管图像级虚拟试穿已取得进展,但扩展到视频时易出现帧间不连续。现有方法通过增加视频块间的重叠来缓解,但导致重复计算,尤其在长视频中成本高昂。为此,我们重新将视频虚拟试穿视为条件视频修复任务,以衣物作为输入条件。具体地,通过引入时序注意力层增强图像扩散模型的时序一致性;提出ShiftCaching技术,在保持一致性的同时大幅减少冗余计算。此外,构建了TikTokDress数据集,包含更复杂背景、挑战性动作和更高分辨率,优于现有公开数据集。大量实验表明,本方法在视频一致性与推理速度上均超越现有基线。项目页面:https://swift-try.github.io/

原文摘要 · Abstract (English)

Given an input video of a person and a new garment, the objective of this paper is to synthesize a new video where the person is wearing the specified garment while maintaining spatiotemporal consistency. Although significant advances have been made in image-based virtual try-on, extending these successes to video often leads to frame-to-frame inconsistencies. Some approaches have attempted to address this by increasing the overlap of frames across multiple video chunks, but this comes at a steep computational cost due to the repeated processing of the same frames, especially for long video sequences. To tackle these challenges, we reconceptualize video virtual try-on as a conditional video inpainting task, with garments serving as input conditions. Specifically, our approach enhances image diffusion models by incorporating temporal attention layers to improve temporal coherence. To reduce computational overhead, we propose ShiftCaching, a novel technique that maintains temporal consistency while minimizing redundant computations. Furthermore, we introduce the TikTokDress dataset, a new video try-on dataset featuring more complex backgrounds, challenging movements, and higher resolution compared to existing public datasets. Extensive experiments demonstrate that our approach outperforms current baselines, particularly in terms of video consistency and inference speed. The project page is available at https://swift-try.github.io/.

视频生成扩散模型虚拟试穿时序一致性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。