用上下文匹配提升人像生成一致性,支持任意姿态与光影重演。
IC-Portrait: In-Context Matching for View-Consistent Personalized Portrait
- 通过掩码80%图像自监督学习光照特征,实现精准光照感知。
- 利用合成视图一致数据集,使参考人像可任意姿态对齐。
- 无需微调,直接拼接潜空间即可提升身份保真度与3D重光能力。
现有扩散模型在保持身份特征方面表现优异,但个性化人像生成仍受用户外貌和光照差异的挑战。为此,我们提出IC-Portrait框架,通过预训练扩散模型在上下文中的快速学习能力(约100~200步)设计两个核心模块:1)光照感知拼接:掩码输入图像80%以上,可有效自监督学习参考图像的光照表征;2)视图一致适配:利用合成视图一致人物数据集,学习上下文对应关系,使参考画像可被扭曲至任意姿态以实现强空间对齐的视图条件控制。通过简单拼接潜空间形成类似ControlNet的监督机制,显著提升身份保真度与稳定性。大量实验表明,IC-Portrait在定量与定性评估中均优于现有最优方法,尤其在视觉质量上表现突出,并展现出3D感知重光能力。
原文摘要 · Abstract (English)
Existing diffusion models show great potential for identity-preserving generation. However, personalized portrait generation remains challenging due to the diversity in user profiles, including variations in appearance and lighting conditions. To address these challenges, we propose IC-Portrait, a novel framework designed to accurately encode individual identities for personalized portrait generation. Our key insight is that pre-trained diffusion models are fast learners (e.g.,100 ~ 200 steps) for in-context dense correspondence matching, which motivates the two major designs of our IC-Portrait framework. Specifically, we reformulate portrait generation into two sub-tasks: 1) Lighting-Aware Stitching: we find that masking a high proportion of the input image, e.g., 80%, yields a highly effective self-supervisory representation learning of reference image lighting. 2) View-Consistent Adaptation: we leverage a synthetic view-consistent profile dataset to learn the in-context correspondence. The reference profile can then be warped into arbitrary poses for strong spatial-aligned view conditioning. Coupling these two designs by simply concatenating latents to form ControlNet-like supervision and modeling, enables us to significantly enhance the identity preservation fidelity and stability. Extensive evaluations demonstrate that IC-Portrait consistently outperforms existing state-of-the-art methods both quantitatively and qualitatively, with particularly notable improvements in visual qualities. Furthermore, IC-Portrait even demonstrates 3D-aware relighting capabilities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。