解决多个Adapter共用扩散模型时的身份混淆问题
SDO: Subspace Deconflicting Operator for Multi-Adapter Composition
- 通过分析参数空间中的子空间冲突，设计去冲突算子
- 在多角色生成中显著提升身份保真度和组合稳定性
- 适合需要组合多个角色的可控图像生成场景
在共享扩散主干网络中组合独立训练的Adapter可实现多角色生成的模块化设计，但直接联合部署常导致身份混杂、跨角色属性泄露和场景不稳定。本文从参数空间角度研究该干扰现象，假设其部分源于共享层中重叠主导子空间的冲突。为此提出SDO（Subspace Deconflicting Operator），通过重构选定Adapter的层级低秩更新，提取紧凑子空间特征，基于输出子空间重叠度衡量成对冲突，并施加置换等变变换以抑制有害共享方向，同时保留身份特异性特征。最终表示映射回标准Adapter更新形式，可直接集成至现有扩散推理流程。实验表明，SDO在多适配器联合生成任务中持续提升身份保真度与组合稳定性，且随着适配器数量增加，性能提升更显著。
原文摘要 · Abstract (English)
Composing independently trained adapters within a shared diffusion backbone provides a modular approach to multi-character generation, but naive joint deployment often causes identity mixing, cross-character attribute leakage, and unstable scene composition. We study this interference from a parameter-space perspective and hypothesize that it arises partly from conflicts between overlapping dominant subspaces in shared layers. To address this issue, we propose \textbf{SDO}, a \textbf{S}ubspace \textbf{D}econflicting \textbf{O}perator for multi-adapter composition. SDO reconstructs layer-wise low-rank updates from the selected adapters, extracts compact subspace signatures, measures pairwise conflict through output-subspace overlap, and applies a permutation-equivariant transformation that suppresses harmful shared directions while retaining identity-specific characteristics. The resulting representations are mapped back to standard adapter updates and can be directly incorporated into existing diffusion inference pipelines. Experiments demonstrate that SDO consistently improves identity fidelity and compositional stability, with particularly clear gains as the number of jointly composed adapters increases.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。