只用0.7%参数，定位关键模块让微调更高效
Rethinking Adapter Placement: A Dominant Adaptation Module Perspective
- 用梯度能量探测器找最敏感的适配模块
- 单个浅层前馈网络模块占主导适配效果
- 适合追求高效微调的开发者和研究者
低秩适配（LoRA）是一种广泛使用的参数高效微调方法，将可训练的低秩适配器插入冻结的预训练模型中。尽管近期研究发现减少适配器数量仍可维持甚至提升性能，但现有方法仍普遍分散放置适配器，如何在有限数量下最大化性能仍不明确。为此，我们提出PAGE（投影适配器梯度能量），一种基于梯度的敏感性探针，用于估计每个候选适配器的初始可训练梯度能量。令人惊讶的是，我们在两个模型家族和四个下游任务中发现，该能量高度集中于单一浅层前馈网络下投影模块。我们称此模块为主导适配模块，并证明其层索引与架构相关但任务稳定。受此发现启发，我们提出DomLoRA，仅在主导适配模块放置一个适配器。仅需约0.7%的原始LoRA可训练参数，DomLoRA在指令遵循、数学推理、代码生成和多轮对话等多种下游任务上平均表现更优。该方法还提升了其他LoRA变体的性能，支持主导适配模块视角作为实用的适配器放置指导。
原文摘要 · Abstract (English)
Low-rank adaptation (LoRA) is a widely used parameter-efficient fine-tuning method that places trainable low-rank adapters into frozen pre-trained models. Recent studies show that using fewer LoRA adapters may still maintain or even improve performance, but existing methods still distribute adapters broadly, leaving where to place a limited number of adapters to maximize performance largely open. To investigate this, we introduce PAGE (Projected Adapter Gradient Energy), a gradient-based sensitivity probe that estimates the initial trainable gradient energy available to each candidate LoRA adapter. Surprisingly, we find that PAGE is highly concentrated on a single shallow FFN down-projection across two model families and four downstream tasks. We term this module the dominant adaptation module and show that its layer index is architecture-dependent but task-stable. Motivated by this finding, we propose DomLoRA, a placement method that places a single adapter at the dominant adaptation module. With only ~0.7% of vanilla LoRA's trainable parameters, DomLoRA outperforms it on average across various downstream tasks, including instruction following, mathematical reasoning, code generation, and multi-turn conversation. This method also improves other LoRA variants, supporting the dominant adaptation module perspective as a practical placement guideline.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。