解决扩散模型多概念生成时的混淆问题，实现精准个性化。
ConceptSplit: Decoupled Multi-Concept Personalization of Diffusion Models via Token-wise Adaptation and Attention Disentanglement
- 通过逐标记值适配，只优化交叉注意力的值投影，避免干扰。
- 在推理时优化潜在表示，显著减少多个概念间的注意力纠缠。
- 适合需要精确控制多个主体形象的图像生成用户。
近年来，文本到图像扩散模型的多概念个性化受到广泛关注，旨在图像中同时表现多个主体。主要挑战是“概念混淆”，即多个学习到的概念在输出图像中产生不期望的干扰或混合。为此，本文提出 ConceptSplit 框架，通过训练与推理阶段分离实现概念解耦。框架包含两个核心组件：首先，提出无合并的训练方法——逐标记值适配（ToVA），仅针对交叉注意力中的值投影进行调整；实证分析表明，修改键投影会破坏注意力机制并引发概念混淆。其次，提出推理阶段的潜在优化方法——解纠缠注意力潜在优化（LODA），通过优化输入潜在变量缓解注意力纠缠。大量定性与定量实验表明，ConceptSplit 能有效实现鲁棒的多概念个性化，显著降低意外概念干扰。代码已公开于 https://github.com/KU-VGI/ConceptSplit。
原文摘要 · Abstract (English)
In recent years, multi-concept personalization for text-to-image (T2I) diffusion models to represent several subjects in an image has gained much more attention. The main challenge of this task is "concept mixing", where multiple learned concepts interfere or blend undesirably in the output image. To address this issue, in this paper, we present ConceptSplit, a novel framework to split the individual concepts through training and inference. Our framework comprises two key components. First, we introduce Token-wise Value Adaptation (ToVA), a merging-free training method that focuses exclusively on adapting the value projection in cross-attention. Based on our empirical analysis, we found that modifying the key projection, a common approach in existing methods, can disrupt the attention mechanism and lead to concept mixing. Second, we propose Latent Optimization for Disentangled Attention (LODA), which alleviates attention entanglement during inference by optimizing the input latent. Through extensive qualitative and quantitative experiments, we demonstrate that ConceptSplit achieves robust multi-concept personalization, mitigating unintended concept interference. Code is available at https://github.com/KU-VGI/ConceptSplit
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。