用合成加真实数据训练,让换妆更保真且不改脸。
From Synthetic to Real: Toward Identity-Consistent Makeup Transfer with Synthetic and Real Data

- 构建合成数据流水线,确保妆容和身份一致
- 引入强化学习适配真实场景,性能显著提升
- 新基准覆盖多元人群,适合实际应用评估
妆容迁移旨在将参考肖像的妆容风格迁移到源肖像上,同时保持身份和背景不变。早期方法将其视为无监督图像到图像翻译,依赖代理目标,性能有限。近期基于扩散和流的方法利用合成数据进行有监督训练,效果显著提升。然而仍面临两大挑战:合成监督常无法忠实保持身份一致性,且合成与真实数据间的域差距限制泛化能力,导致复杂真实场景下性能下降。为此,本文提出ConsistentBeauty,一种新颖的数据清洗流程,确保合成数据中妆容保真度与身份严格一致;其次提出RealBeauty,一种从合成到真实的后训练框架。在高质量合成数据监督基础上,通过强化学习进一步适应真实场景,并设计可验证的奖励函数,使模型能吸收真实妆容模式。此外,建立新多样性基准,涵盖广泛肤色、年龄、性别、姿态和妆容风格,支持更全面的真实世界评估。大量实验表明,该方法在多个基准上达到顶尖性能,尤其在身份保护和复杂真实案例中优势明显。
原文摘要 · Abstract (English)
Makeup transfer aims to apply the makeup style of a reference portrait to a source portrait while preserving identity and background. Early methods formulate this task as unsupervised image-to-image translation, relying on surrogate objectives and often yielding limited performance. Recent diffusion- and flow-based approaches instead exploit synthetic data for supervised training, leading to significant improvements. However, these methods still face two critical challenges: synthetic supervision frequently fails to faithfully preserve identity, and the domain gap between synthetic and real data limits generalization, resulting in degraded performance in complex real-world scenarios. To address these issues, this paper first proposes ConsistentBeauty, a novel data curation pipeline that ensures makeup fidelity and strict identity consistency within the synthesized data. Second, we propose RealBeauty, a synthetic-to-real post-training framework. Beyond supervised learning on curated synthetic data, we further adapt the model to real-world scenarios through reinforcement learning and design novel verifiable rewards tailored to the makeup transfer task. It allows the model to further benefit from real makeup patterns beyond synthetic supervision. In addition, we establish a new diverse benchmark for makeup transfer, covering a wide range of skin tones, ages, genders, poses, and makeup styles, thereby enabling a more comprehensive evaluation of model performance under diverse real-world conditions. Extensive experiments show that our method achieves state-of-the-art performance on multiple benchmarks and demonstrates clear advantages in identity preservation and performance on complex real-world cases.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。