arXiv:2606.31717cs.LG2026-06

针对自门控网络的低秩微调,提出感知非线性的新方法。

Nonlinearity-Aware LoRA: Structured Gate Adaptation under Low-Rank Constraints

论文配图:Nonlinearity-Aware LoRA: Structured Gate Adaptation under Low-Rank Constraints
图 1 · 摘自论文原文
  • 基于门控通道响应性设计动态更新机制
  • 在语言和视觉-语言任务上优于普通LoRA
  • 无需额外损失函数,推理无开销

低秩适应(LoRA)通常被视为全量微调的更新空间近似,但在自门控Transformer前馈网络中这一观点不完整。在门控前馈网络中,低秩残差不仅改变投影特征,还影响决定哪些通道贡献输出的非线性选择权重。我们将其效应形式化为选择错位,并与自门控激活的局部有效同质性相联系。这启发了一种非线性感知的参数高效微调原则:低秩更新应分配容量给仍具响应性的门控通道,并调控选择的时序演化。我们提出NA-LoRA,一种仅训练阶段使用的轻量级方法,包含两个机制:基于导数的时间重要性掩码用于门相关LoRA更新,以及在存在有意义的粗粒度有效同质性划分时的激活特定步长缩放规则。NA-LoRA不添加辅助损失,且推理时无额外开销。在语言模型微调和视觉-语言迁移基准上的实验表明,NA-LoRA持续优于普通LoRA,且与强性能的PEFT变体相当或更优。

原文摘要 · Abstract (English)

Low-rank adaptation (LoRA) is commonly viewed as an update-space approximation to full fine-tuning, yet this view is incomplete for self-gated Transformer feed-forward networks. In gated FFNs, a low-rank residual can change not only projected features but also the nonlinear selection weights that determine which channels contribute to the output. We formalize this effect as selection misalignment and connect it to the local effective homogeneity of self-gated activations. This motivates a nonlinearity-aware principle for parameter-efficient fine-tuning: low-rank updates should allocate capacity to gate channels whose nonlinear states remain responsive and should shape the temporal evolution of selection. We propose NA-LoRA, a training-only method with two lightweight mechanisms: a derivative-based temporal-importance mask for gate-related LoRA updates and an activation-specific step-scaling rule when a meaningful coarse effective-homogeneity partition is available. NA-LoRA adds no auxiliary loss and incurs no inference-time overhead. Experiments on language-model fine-tuning and vision-language transfer benchmarks show that NA-LoRA consistently improves over vanilla LoRA and is competitive with or better than strong PEFT variants.

LoRA非线性微调

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。