让任意人物无缝替换进视频,保持身份、动作与环境一致
SwapAnyone: Consistent and Realistic Video Synthesis for Swapping Any Person into Any Video
- 将人体替换视为端到端视频修复任务,引入三重一致性约束
- 在多个指标上超越开源方法,接近甚至超过闭源模型表现
- 适用于影视剪辑、虚拟演出等需要真实感人物替换的场景
视频人体替换旨在将现有视频中的身体替换成任意来源的新身体,近年来受到广泛关注。现有方法将该任务拆分为多个子任务,依赖多个模型顺序处理,缺乏端到端优化,导致帧间亮度不一致、遮挡关系混乱、人物与背景分离等问题。本文将视频人体替换定义为独立任务,提出身份一致性、运动一致性与环境一致性三重约束。提出端到端模型SwapAnyone,将其视为带参考保真度与运动控制的视频修复任务。为提升环境和谐性(尤其亮度一致性),设计渐进式EnvHarmony训练策略。构建包含32,000个视频的人体动作数据集HumanAction-32K。大量实验表明,本方法在开源方案中达到最先进水平,多项指标接近或超越闭源模型。所有代码、模型权重及数据集将开源至https://github.com/PKU-YuanGroup/SwapAnyone。
原文摘要 · Abstract (English)
Video body-swapping aims to replace the body in an existing video with a new body from arbitrary sources, which has garnered more attention in recent years. Existing methods treat video body-swapping as a composite of multiple tasks instead of an independent task and typically rely on various models to achieve video body-swapping sequentially. However, these methods fail to achieve end-to-end optimization for the video body-swapping which causes issues such as variations in luminance among frames, disorganized occlusion relationships, and the noticeable separation between bodies and background. In this work, we define video body-swapping as an independent task and propose three critical consistencies: identity consistency, motion consistency, and environment consistency. We introduce an end-to-end model named SwapAnyone, treating video body-swapping as a video inpainting task with reference fidelity and motion control. To improve the ability to maintain environmental harmony, particularly luminance harmony in the resulting video, we introduce a novel EnvHarmony strategy for training our model progressively. Additionally, we provide a dataset named HumanAction-32K covering various videos about human actions. Extensive experiments demonstrate that our method achieves State-Of-The-Art (SOTA) performance among open-source methods while approaching or surpassing closed-source models across multiple dimensions. All code, model weights, and the HumanAction-32K dataset will be open-sourced at https://github.com/PKU-YuanGroup/SwapAnyone.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。