arXiv:2606.03792cs.CVcs.LG2026-06中稿 · IEEE FG 2026

无需训练即可实现多概念图像生成,通过提示词权重自动调节不同LoRA模块贡献。

Training-Free Multi-Concept LoRA Composition with Prompt-Aware Weighting

论文配图:Training-Free Multi-Concept LoRA Composition with Prompt-Aware Weighting
图 1 · 摘自论文原文
  • 根据提示词中关键词的语义重要性,动态加权多个LoRA模块输出。
  • 在ComposLoRA测试集上,生成图像质量、身份保留与组合能力均优于现有方法。
  • 适合需要快速组合多个视觉概念且不希望重新训练的用户使用。

低秩适应(LoRA)通过微调预训练扩散模型,实现了文本到图像生成中的个性化。然而,将该技术扩展至多概念定制仍具挑战性。简单合并多个LoRA权重或输出常导致概念间干扰,降低视觉质量并削弱对参考图像的保真度。本文提出一种无需训练的多概念定制方法,通过最优组合多个LoRA模块的输出来解决此问题。我们基于提示词中对应概念的相对重要性,引入两种方法:W-Switch与W-Composite,依据触发词在目标提示中的语义影响动态加权各LoRA。同时,我们拓展了现有评估指标,提出一种基于图像的相似性评估框架,通过对比真实参考图像与生成图像中自动分割的概念区域,评估图像保真度与身份保持能力。在ComposLoRA测试平台上,我们的方法在视觉质量、身份保留和组合性方面持续优于当前最优方法。定性评估包括基于大语言模型的分析和用户研究,结果与新提出的定量指标一致。代码已公开于https://github.com/GeorgeTsoumplekas/Prompt-Aware-Multi-LoRA-Composition。

原文摘要 · Abstract (English)

Low-Rank Adaptation (LoRA) successfully enables personalization in text-to-image generation by adapting pre-trained diffusion models to specific visual concepts and styles. However, extending such models to multi-concept customization remains challenging. Naively combining multiple LoRA weights or their outputs often leads to interference among concepts, resulting in degraded visual quality and reduced fidelity to the reference images of individual concepts. This paper proposes a simple yet effective approach for multi-concept customization by optimally combining the outputs of multiple LoRA modules. We leverage the relative importance of each concept during generation, as inferred from its corresponding prompt tokens and introduce two methods, W-Switch and W-Composite, that employ a prompt-aware importance weighting strategy in which each LoRA is weighted according to the semantic influence of its trigger words in the target prompt. In addition, we extend existing quantitative evaluation metrics by proposing a new image-based similarity evaluation framework that assesses image fidelity and identity preservation through comparisons between real-world reference images and automatically segmented concept regions from generated images. We evaluate our approach on the ComposLoRA testbed and demonstrate consistent improvements over existing state-of-the-art methods in terms of visual quality, identity preservation and compositionality. Qualitative evaluations, including a Large Language Model (LLM) based assessment and a user study, further validate the effectiveness of the proposed methods and align with the newly introduced quantitative image-based metrics. Our code is available at https://github.com/GeorgeTsoumplekas/Prompt-Aware-Multi-LoRA-Composition.

LoRA图像生成多概念提示词加权

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。