提出BlockLoRA，可快速融合15个概念且不丢失原身份。
Modular Customization of Diffusion Models via Blockwise-Parameterized Low-Rank Adaptation
- 分块低秩参数化+随机输出擦除，减少概念干扰。
- 一次合并15个概念，保持身份清晰度高。
- 适合多用户协作、风格与主体混合定制场景。
近期扩散模型定制已能用少量图像融入特定主题或风格概念，但多个概念的模块化组合仍待解决——即在不破坏原有身份的前提下，高效融合由不同用户训练的独立概念。现有方法或仅支持固定概念集需重新训练，或虽可即时合并却易导致身份混淆与干扰，且通常限于少量概念。为此，本文提出BlockLoRA，一种即时融合方法，通过分析干扰根源，引入随机输出擦除技术以最小化不同定制模型间的干扰，并采用分块低秩参数化策略降低合并过程中的身份损失。大量实验验证，BlockLoRA可高保真地即时融合15个概念（人物、主题、场景、风格），有效保留各概念独立性，适用于跨用户协作与多概念复合定制任务。
原文摘要 · Abstract (English)
Recent diffusion model customization has shown impressive results in incorporating subject or style concepts with a handful of images. However, the modular composition of multiple concepts into a customized model, aimed to efficiently merge decentralized-trained concepts without influencing their identities, remains unresolved. Modular customization is essential for applications like concept stylization and multi-concept customization using concepts trained by different users. Existing post-training methods are only confined to a fixed set of concepts, and any different combinations require a new round of retraining. In contrast, instant merging methods often cause identity loss and interference of individual merged concepts and are usually limited to a small number of concepts. To address these issues, we propose BlockLoRA, an instant merging method designed to efficiently combine multiple concepts while accurately preserving individual concepts' identity. With a careful analysis of the underlying reason for interference, we develop the Randomized Output Erasure technique to minimize the interference of different customized models. Additionally, Blockwise LoRA Parameterization is proposed to reduce the identity loss during instant model merging. Extensive experiments validate the effectiveness of BlockLoRA, which can instantly merge 15 concepts of people, subjects, scenes, and styles with high fidelity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。