通过投影抑制风格干扰,实现内容与风格的精准融合。
NP-LoRA: Null Space Projection for Subject-Style LoRA Fusion
- 用主方向投影将内容LoRA映射到风格的补空间,减少冲突。
- 在多个预训练组合上,生成质量优于强基线,无需重新训练。
- 适合需要高效可控生成的视频/图像创作场景。
低秩适配(LoRA)融合可实现无需重训练的内容与风格组合生成。然而,现有方法主要基于权重级合并,未显式建模独立训练的LoRA在共享参数空间中的交互方式。本文从几何视角出发,将内容与风格LoRA视为占据重叠但非正交的低秩子空间,其重叠会导致参数更新冲突,影响生成质量。为此,我们提出训练自由的空投影LoRA(NP-LoRA),以投影作为融合算子,显式调控跨LoRA交互。具体地,利用风格LoRA的主方向定义投影子空间,将内容LoRA投影至其补空间(即风格的零空间),抑制沿主导风格方向的干扰,同时保留互补信息。为避免硬投影过于激进,进一步将软投影建模为带正则化的优化问题,平衡内容保留与风格子空间抑制,该目标具有闭式解,仅需单个参数控制,连续介于线性合并与硬投影之间。在多个预训练LoRA对上的实验表明,NP-LoRA实现了更均衡的内容-风格组合,且无需重训练。
原文摘要 · Abstract (English)
Low-Rank Adaptation (LoRA) fusion enables the composition of subject and style representations for controllable generation without retraining. However, existing approaches primarily operate through weight-level merging, without explicitly modeling how independently trained LoRAs interact in the shared parameter space. We adopt a geometric perspective on LoRA fusion, interpreting content and style LoRAs as occupying overlapping, non-orthogonal low-rank subspaces, where such overlap can lead to conflicting parameter updates that affect generation quality. This observation motivates us to reformulate LoRA fusion not merely as parameter combination, but as a problem of controlling how updates from overlapping subspaces are combined. Based on this insight, we propose Null Space Projection LoRA (NP-LoRA), a training-free framework that employs projection as a fusion operator to explicitly modulate cross-LoRA interactions. Specifically, NP-LoRA uses principal directions of the style LoRA to define a projection subspace and projects the content LoRA onto the complementary subspace (i.e., the null space of the style LoRA), suppressing interference along dominant style directions while preserving complementary information. To avoid the overly aggressive suppression of hard projection, we further formulate soft projection as a regularized optimization problem that balances content preservation against style-subspace suppression. This objective admits a closed-form solution, yielding a projection operator controlled by a single parameter that continuously interpolates between linear merging and hard projection. Extensive experiments across multiple pretrained LoRA pairs show that NP-LoRA achieves more balanced content-style composition compared to strong baselines, without requiring retraining.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。