给普通人脸生成专属名字，让扩散模型稳定识别身份。
MagicNaming: Consistent Identity Generation by Finding a "Name Space" in T2I Diffusion Models
- 从名人名字嵌入中构建‘名字空间’，映射人脸特征
- 用预测的名字嵌入提升生成图像的身份一致性
- 无需重训练，适配各类SDXL衍生模型
大规模文生图扩散模型（如DALL-E、SDXL）可通过姓名生成名人图像。本文探索能否像名人一样，仅用名字生成通用身份。研究发现，在扩散模型文本编码器的名人名字嵌入空间中存在一个“名字空间”，其中每个点对应特定身份。我们首先提取Laion5B数据集中名人名字的文本嵌入，作为监督信号，训练一个编码器，将给定人脸图像映射到对应名字嵌入。实验表明，该名字嵌入能有效保证生成图像的身份一致性。这些嵌入与文本语义解耦，不干扰原模型的生成能力。只需简单插入此类嵌入，所有基于SDXL的变体（如Civitai中的模型）即可变为具备身份感知能力的文生图模型。
原文摘要 · Abstract (English)
Large-scale text-to-image diffusion models, (e.g., DALL-E, SDXL) are capable of generating famous persons by simply referring to their names. Is it possible to make such models generate generic identities as simple as the famous ones, e.g., just use a name? In this paper, we explore the existence of a "Name Space", where any point in the space corresponds to a specific identity. Fortunately, we find some clues in the feature space spanned by text embedding of celebrities' names. Specifically, we first extract the embeddings of celebrities' names in the Laion5B dataset with the text encoder of diffusion models. Such embeddings are used as supervision to learn an encoder that can predict the name (actually an embedding) of a given face image. We experimentally find that such name embeddings work well in promising the generated image with good identity consistency. Note that like the names of celebrities, our predicted name embeddings are disentangled from the semantics of text inputs, making the original generation capability of text-to-image models well-preserved. Moreover, by simply plugging such name embeddings, all variants (e.g., from Civitai) derived from the same base model (i.e., SDXL) readily become identity-aware text-to-image models. Project homepage: \url{https://magicfusion.github.io/MagicNaming/}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。