arXiv:2508.08812cs.CV2025-08AAAI被引 4

TARA让扩散模型轻松组合多个个性化概念,不打架、不串味。

TARA: Token-Aware LoRA for Composable Personalization in Diffusion Models

  • 用令牌掩码让每个LoRA专注对应关键词,避免干扰
  • 训练时对齐罕见词注意力与概念区域,提升定位精度
  • 无需重训,直接注入多个模块就能合成多概念图像

个性化文生图旨在仅用几张参考图生成特定主体或风格的新图像。基于低秩适应(LoRA)的近期方法通过向预训练扩散模型注入轻量级、概念专属适配器,实现高效单概念定制。然而,组合多个LoRA模块进行多概念生成时常导致身份丢失和视觉特征泄露。本文识别出两个关键问题:(1) 不同LoRA模块间的令牌级干扰;(2) 罕见词注意力图与其对应概念区域的空间错位。为此,提出令牌感知LoRA(TARA),引入令牌掩码显式约束每模块聚焦于其关联的罕见词,以避免干扰,并设计训练目标促使罕见词的空间注意力与概念区域对齐。本方法可在推理时直接注入多个独立训练的TARA模块,实现无需重训的多概念组合生成。实验表明,TARA能高效实现多概念推理,并有效保持各概念的视觉身份,避免模块间相互干扰。代码与模型已开源。

原文摘要 · Abstract (English)

Personalized text-to-image generation aims to synthesize novel images of a specific subject or style using only a few reference images. Recent methods based on Low-Rank Adaptation (LoRA) enable efficient single-concept customization by injecting lightweight, concept-specific adapters into pre-trained diffusion models. However, combining multiple LoRA modules for multi-concept generation often leads to identity missing and visual feature leakage. In this work, we identify two key issues behind these failures: (1) token-wise interference among different LoRA modules, and (2) spatial misalignment between the attention map of a rare token and its corresponding concept-specific region. To address these issues, we propose Token-Aware LoRA (TARA), which introduces a token mask to explicitly constrain each module to focus on its associated rare token to avoid interference, and a training objective that encourages the spatial attention of a rare token to align with its concept region. Our method enables training-free multi-concept composition by directly injecting multiple independently trained TARA modules at inference time. Experimental results demonstrate that TARA enables efficient multi-concept inference and effectively preserving the visual identity of each concept by avoiding mutual interference between LoRA modules. The code and models are available at https://github.com/YuqiPeng77/TARA.

扩散模型个性化生成LoRA多概念组合

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。