解决多模态模型修图时人脸身份不一致的问题
Optimizing ID Consistency in Multimodal Large Models: Facial Restoration via Alignment, Entanglement, and Disentanglement
- 通过对齐、解耦与选择性耦合,优化跨源特征融合
- 在真实人脸编辑中实现身份与编辑内容双重一致性
- 无需训练,可直接插入现有模型,适合实际应用
多模态编辑大模型在各类任务中展现出强大能力,但在真实人像编辑中普遍存在面部身份(ID)一致性下降问题。由于人眼对脸部特征高度敏感,这种不一致严重制约了模型的实际应用。现有身份保留方法难以同时保持原始人脸身份和编辑元素的完整性,主要受跨源分布偏移和特征污染影响。为此,我们提出EditedID框架,采用对齐-解耦-耦合机制,通过分析扩散轨迹、采样器行为和注意力特性,引入三项关键技术:1)自适应混合策略,在扩散过程中对齐跨源潜在表示;2)混合求解器,解耦源特定的身份属性与细节;3)注意力门控机制,选择性耦合视觉元素。大量实验表明,EditedID在保持原始人脸身份和编辑内容一致性方面达到当前最优水平。作为无训练、即插即用的解决方案,它为开放世界下单/多人脸身份恢复树立了新基准,推动多模态编辑模型在真人编辑场景中的实际部署。代码已公开于 https://github.com/NDYBSNDY/EditedID。
原文摘要 · Abstract (English)
Multimodal editing large models have demonstrated powerful editing capabilities across diverse tasks. However, a persistent and long-standing limitation is the decline in facial identity (ID) consistency during realistic portrait editing. Due to the human eye's high sensitivity to facial features, such inconsistency significantly hinders the practical deployment of these models. Current facial ID preservation methods struggle to achieve consistent restoration of both facial identity and edited element IP due to Cross-source Distribution Bias and Cross-source Feature Contamination. To address these issues, we propose EditedID, an Alignment-Disentanglement-Entanglement framework for robust identity-specific facial restoration. By systematically analyzing diffusion trajectories, sampler behaviors, and attention properties, we introduce three key components: 1) Adaptive mixing strategy that aligns cross-source latent representations throughout the diffusion process. 2) Hybrid solver that disentangles source-specific identity attributes and details. 3) Attentional gating mechanism that selectively entangles visual elements. Extensive experiments show that EditedID achieves state-of-the-art performance in preserving original facial ID and edited element IP consistency. As a training-free and plug-and-play solution, it establishes a new benchmark for practical and reliable single/multi-person facial identity restoration in open-world settings, paving the way for the deployment of multimodal editing large models in real-person editing scenarios. The code is available at https://github.com/NDYBSNDY/EditedID.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。