提出跨分支冲突防护机制,有效防止统一多模态模型篡改人脸身份。
Cross-Branch Conflict as a Shield: Safeguarding Facial Identities in Unified Multimodal Image Editing

- 通过干扰视觉与生成双分支结构一致性实现防御
- 在多个数据集上使身份保留编辑率下降超过60%
- 适合关注隐私保护的图像编辑研究者使用
统一多模态模型(UMMs)虽具备强大的指令式图像编辑能力,但引发个人肖像被未经授权修改的严重担忧。本文研究如何保护人脸身份免受此类编辑攻击。现有基于扩散模型和视觉语言模型的防护方法常因仅破坏单一视觉分支而失效。我们对统一图像编辑模型的理解与生成分支进行特征级分析,发现两分支结构一致性直接影响编辑成功度。当仅一个分支被扰动时,模型仍可从另一分支恢复身份信息。为此,我们提出跨分支冲突作为盾牌(CCS),一种统一的对抗性防护框架。CCS同时将ViT与VAE表示推向其原始状态的偏离方向,并引入线性中心核对齐(CKA)目标,破坏两分支间结构一致性。通过在双视觉路径中降级可靠的身份信息并诱导不兼容的跨分支表示,CCS有效阻止了模型在编辑过程中恢复一致的人脸身份线索。大量实验表明,CCS在抑制保留身份的编辑方面持续提供更强保护。代码见补充材料。
原文摘要 · Abstract (English)
Unified multimodal models (UMMs) have recently demonstrated powerful instruction-based image editing capabilities, while also raising serious concerns about the unauthorized manipulation of personal portraits. We investigate a novel and practical problem: protecting facial identities against unauthorized editing of UMMs. Existing diffusion-based and VLM-based protection methods often become ineffective because they typically disrupt only a single visual branch. To understand this limitation, we conduct a feature-level analysis of the understanding and generation branches in unified image editing models. Our observations show that the structural agreement between these two branches is closely related to successful image editing. When only one branch is distorted, the model may still recover identity information from the other branch. Based on this, we propose Cross-Branch Conflict as a Shield (CCS), a unified adversarial protection framework. CCS jointly drives the ViT and VAE representations away from their clean counterparts. It also uses a linear Centered Kernel Alignment (CKA) objective to disrupt the structural consistency between the two branches. By degrading reliable identity information in both visual pathways and inducing incompatible cross-branch representations, CCS effectively prevents UMMs from recovering consistent facial identity cues during editing. Extensive experiments suggest that CCS consistently provides stronger protection in suppressing identity-preserving edits. Codes are in the supplementary material.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。