让AI生成图更听话，精准融合个人图片与文字描述。
Conceptrol: Concept Control of Zero-shot Personalized Image Generation
- 用文本概念掩码约束视觉注意力，提升个性化控制力。
- 在基准测试中比原版IP-Adapter提升89%生成准确性。
- 无需微调，适合快速生成符合指令的个性化图像。
基于文本到图像扩散模型的个性化图像生成可在无需训练的情况下依据参考图像内容生成新图像。零样本适配方法如IP-Adapter和OminiControl尤其吸引人，因其无需测试时微调。然而，它们难以平衡保留个性化内容与遵循文本提示之间的关系。我们识别出关键设计缺陷：现有适配器未能有效整合个性化图像与文本描述。因此生成图像更偏向复制个性化内容而非遵循文本指令。但基础文本到图像模型具备强大的概念理解能力，可被利用。我们提出Conceptrol，一种简单高效、无额外计算开销的框架，通过文本概念掩码约束视觉规范的注意力，显著提升以主体为中心的生成能力。该方法在个性化基准上相较原始IP-Adapter最高提升89%，甚至超过如Dreambooth LoRA等微调方法。源代码已公开于https://github.com/QY-H00/Conceptrol。
原文摘要 · Abstract (English)
Personalized image generation with text-to-image diffusion models generates unseen images based on reference image content. Zero-shot adapter methods such as IP-Adapter and OminiControl are especially interesting because they do not require test-time fine-tuning. However, they struggle to balance preserving personalized content and adherence to the text prompt. We identify a critical design flaw resulting in this performance gap: current adapters inadequately integrate personalization images with the textual descriptions. The generated images, therefore, replicate the personalized content rather than adhere to the text prompt instructions. Yet the base text-to-image has strong conceptual understanding capabilities that can be leveraged. We propose Conceptrol, a simple yet effective framework that enhances zero-shot adapters without adding computational overhead. Conceptrol constrains the attention of visual specification with a textual concept mask that improves subject-driven generation capabilities. It achieves as much as 89% improvement on personalization benchmarks over the vanilla IP-Adapter and can even outperform fine-tuning approaches such as Dreambooth LoRA. The source code is available at https://github.com/QY-H00/Conceptrol.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。