arXiv:2506.13479cs.CLcs.AI2025-06

LoRA复用效果存疑,需先搞清其适用条件

Position: Pause Recycling LoRAs and Prioritize Mechanisms to Uncover Limits and Effectiveness

  • 通过理论分析与合成任务检验复用机制
  • 发现跨数据集知识整合常失败,尤其预训练不足时
  • 建议暂停新算法研发,转而建立评估标准

低秩适配器(LoRAs)合并或路由已成为提升大模型性能的热门方案,尤其在数据受限的监管或领域场景下。本文主张研究重点应从开发新合并/路由算法转向理解复用LoRAs的实际有效条件。通过理论分析及合成双跳推理与数学应用题任务,我们检验了复用是否实现真正的组合泛化,还是仅依赖浅层模式匹配。评估两种无数据依赖方法——参数平均与动态适配器选择,发现当知识在预训练中未充分覆盖时,复用LoRAs往往无法逻辑整合来自不相交微调数据集的知识。实证结果结合对LoRA表达能力局限性的理论洞察,揭示了复用在未知任务上的前提条件与约束,质疑其作为真正‘无数据’方法的可行性。我们呼吁暂停对新型复用方法的追求,强调需建立严谨机制以指导未来基于适配器的模型合并研究,并为从业者提供实践设计框架。

原文摘要 · Abstract (English)

Merging or routing low-rank adapters (LoRAs) has emerged as a popular solution for enhancing large language models, particularly when data access is restricted by regulatory or domain-specific constraints. This position paper argues that the research community should shift its focus from developing new merging or routing algorithms to understanding the conditions under which reusing LoRAs is truly effective. Through theoretical analysis and synthetic two-hop reasoning and math word-problem tasks, we examine whether reusing LoRAs enables genuine compositional generalization or merely reflects shallow pattern matching. Evaluating two data-agnostic methods--parameter averaging and dynamic adapter selection--we found that reusing LoRAs often fails to logically integrate knowledge across disjoint fine-tuning datasets, especially when such knowledge is underrepresented during pretraining. Our empirical results, supported by theoretical insights into LoRA's limited expressiveness, highlight the preconditions and constraints of reusing them for unseen tasks and cast doubt on its feasibility as a truly data-free approach. We advocate for pausing the pursuit of novel methods for recycling LoRAs and emphasize the need for rigorous mechanisms to guide future academic research in adapter-based model merging and practical system designs for practitioners.

LoRA模型融合可解释性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。