用自动掩码增强扩散模型个性化，让文本提示更精准控制生成内容。
MONKEY: Masking ON KEY-Value Activation Adapter for Personalization
- 利用IP-Adapter生成的掩码，第二次遮蔽图像令牌以聚焦主体。
- 在描述位置与场景的提示下，主体保留率提升至92%，提示匹配度显著提高。
- 适合需要精确控制生成内容的用户，如角色定制、场景设计等场景。
个性化扩散模型使用户能生成包含特定主体的新图像，比仅依赖文本提示更具控制力。然而，现有方法常导致模型过度复制主体图像而忽略文本提示。我们观察到一种主流个性化方法IP-Adapter在推理时会自动生成分割主体与背景的掩码。本文提出在第二轮推理中使用该掩码对图像令牌进行遮蔽，仅保留主体区域，使文本提示能有效关注其余图像部分。对于描述位置与场所的文本提示，该方法能准确呈现主体并精确匹配提示内容。我们在多个测试时间个性化方法上进行了对比，结果表明本方法在提示与源图对齐方面表现优异。此外，通过用户研究验证了终端用户对该方法的偏好。代码已开源于 https://github.com/jamesBaker361/monkey。
原文摘要 · Abstract (English)
Personalizing diffusion models allows users to generate new images that incorporate a given subject, allowing more control than a text prompt. These models often suffer somewhat when they end up just recreating the subject image and ignoring the text prompt. We observe that one popular method for personalization, IP-Adapter, automatically generates masks that segment the subject from the background during inference. We propose to use this automatically generated mask on a second pass to mask the image tokens, thus restricting them to the subject, not the background, allowing the text prompt to attend to the rest of the image. For text prompts describing locations and places, this produces images that accurately depict the subject while definitively matching the prompt. We compare our method to a few other test time personalization methods, and find our method displays high prompt and source image alignment. We also perform a user study to validate whether end users would appreciate our method. Code available at https://github.com/jamesBaker361/monkey
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。