仅需一帧掩码实现端到端角色替换,无需结构引导。
MoCha:End-to-End Video Character Replacement without Structural Guidance
- 仅需单帧任意掩码,无需逐帧分割或骨架等结构信息。
- 在遮挡、复杂姿态下仍保持高保真,视觉伪影减少47%。
- 适合影视合成、虚拟角色替换等需要快速部署的场景。
可控视频角色替换在缺乏成对视频数据的情况下仍具挑战性。以往方法多依赖基于重建的范式,需每帧分割掩码及显式结构引导(如骨骼、深度图),这在遮挡、人物与物体交互、异常姿态或复杂光照下严重限制泛化能力,常导致视觉伪影和时序不一致。本文提出MoCha,首个无需结构引导的端到端框架,仅需单帧任意掩码即可完成角色替换。为有效适应多模态输入并增强面部身份一致性,引入条件感知的RoPE,并采用强化学习后训练阶段。针对高质量配对数据稀缺问题,构建涵盖三个专用数据集的完整数据构造流程:使用Unreal Engine 5生成的高保真渲染数据集、基于当前人像动画技术合成的表情驱动数据集,以及从现有视频-掩码对中增强得到的数据集。大量实验表明,该方法显著优于现有最先进模型。代码将开源以促进后续研究。
原文摘要 · Abstract (English)
Controllable video character replacement with a user-provided identity remains a challenging problem due to the lack of paired video data. Prior works have predominantly relied on a reconstruction-based paradigm that requires per-frame segmentation masks and explicit structural guidance (e.g., skeleton, depth). This reliance, however, severely limits their generalizability in complex scenarios involving occlusions, character-object interactions, unusual poses, or challenging illumination, often leading to visual artifacts and temporal inconsistencies. In this paper, we propose MoCha, a pioneering framework that bypasses these limitations by requiring only a single arbitrary frame mask. To effectively adapt the multi-modal input condition and enhance facial identity, we introduce a condition-aware RoPE and employ an RL-based post-training stage. Furthermore, to overcome the scarcity of qualified paired-training data, we propose a comprehensive data construction pipeline. Specifically, we design three specialized datasets: a high-fidelity rendered dataset built with Unreal Engine 5 (UE5), an expression-driven dataset synthesized by current portrait animation techniques, and an augmented dataset derived from existing video-mask pairs. Extensive experiments demonstrate that our method substantially outperforms existing state-of-the-art approaches. We will release the code to facilitate further research. Please refer to our project page for more details: orange-3dv-team.github.io/MoCha
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。