提出新方法让多个LoRA模块合并时互不干扰，但发现这不等于语义解耦。
Rethinking Inter-LoRA Orthogonality in Adapter Merging: Insights from Orthogonal Monte Carlo Dropout
- 用随机丢弃法强制多个LoRA模块的语义向量严格正交。
- 合并后模块间无直接干扰，但无法实现语义成分化。
- 适用于需要稳定组合多个功能模块的研究者。
我们提出正交蒙特卡洛丢弃（Orthogonal Monte Carlo Dropout），一种在不增加额外时间复杂度的情况下，强制合并稀疏语义向量保持严格正交性的机制。低秩适配（LoRA）是一种流行的大型模型微调方法，通常训练模块以表示特定概念，如物体或风格。当多个LoRA模块合并以生成特定风格的物体时，其输出（语义向量）可能相互干扰。我们的方法确保合并后的LoRA模块保持正交，从而避免直接干扰。然而，实证分析表明，这种正交性并不能带来先前研究中强调的组合式适配中的语义解耦。这一发现提示，仅靠跨LoRA正交性可能不足以实现真正的语义组合性，因而需要重新审视其在适配器合并中的作用。
原文摘要 · Abstract (English)
We propose Orthogonal Monte Carlo Dropout, a mechanism that enforces strict orthogonality when combining sparse semantic vectors without extra time complexity. Low-Rank Adaptation (LoRA), a popular fine-tuning method for large models, typically trains a module to represent a specific concept such as an object or a style. When multiple LoRA modules are merged, for example to generate an object in a particular style, their outputs (semantic vectors) may interfere with each other. Our method guarantees that merged LoRA modules remain orthogonal and thus free from direct interference. However, empirical analysis reveals that such orthogonality does not lead to the semantic disentanglement highlighted in prior work on compositional adaptation. This finding suggests that inter-LoRA orthogonality alone may be insufficient for achieving true semantic compositionality, prompting a re-examination of its role in adapter merging.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。