发现适配器干扰主要源于非线性表示交互，而非参数空间几何。
PermDoRA -- Understanding Adapter Interference in Language Models: Limits of Parameter-Space Geometry
- 用黎曼几何思路优化适配器融合方式
- 多领域表现未优于传统平均融合方法
- 适配器方向正交性对性能预测力弱
大型语言模型的访问控制需模块化机制实现领域特定行为，避免重训练与跨域干扰。现有假设认为适配器组合时的干扰源于线性参数更新的重叠，因此应通过强制正交或方向独立提升多领域性能。本文基于权重分解低秩适配（DoRA-RBAC）框架，在LLaMA-3.1-8B和Mistral-7B上对比传统欧氏融合与一种基于黎曼几何的弗雷歇均值近似融合策略，评估其在GPQA、PubMedQA、SimpleQA、WMDP等问答基准上的表现。结果表明：尽管单领域性能与LoRA相当，但几何感知融合在多领域场景中并无稳定优势；诊断分析进一步显示，适配器更新的夹角对齐与正交性无法有效预测组合性能。这表明适配器干扰主要由共享非线性表示中的交互决定，而非参数空间几何结构。
原文摘要 · Abstract (English)
Access control in large language models (LLMs) requires modular mechanisms to enable domain-specific behavior without retraining or cross-domain interference. A common hypothesis is that interference during adapter composition arises from overlap in linear parameter updates, suggesting that enforcing orthogonality or directional independence should improve multi-domain performance. We test this hypothesis using DoRA-RBAC, a hierarchical adapter composition framework based on weight-decomposed low-rank adaptation. We compare conventional Euclidean merging with a geometry-aware Riemannian-inspired merging strategy that approximates the Frechet mean via normalized directional averaging across multiple QA benchmarks (GPQA, PubMedQA, SimpleQA, WMDP) on LLaMA-3.1-8B and Mistral-7B. Our results show that while single-domain performance matches LoRA, geometry-aware merging provides no consistent advantage over standard averaging in multi-domain settings.Diagnostic analysis further reveals that angular alignment and orthogonality of adapter updates are weak predictors of composition performance. These findings suggest that adapter interference is not governed primarily by parameter-space geometry, but is instead consistent with interactions in shared nonlinear representations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。