通过激活边界匹配,提升LoRA微调初始化效果。
Activation Boundary Matching: Task-Informed Initialization for Low-Rank Adaptation
- 用早期任务探针提取激活符号作为目标,指导新适配器初始化。
- 仅需少量前向计算,性能接近全量训练参考模型。
- 适合追求高效微调的NLP与视觉任务研究者。
低秩适应(LoRA)对初始化敏感,现有方法基于预训练点统计构建初始子空间,捕捉的是预适应几何而非学习过程中的移动方向。我们分析了早期微调轨迹,发现任务诱导的激活边界(层间预激活符号)恢复速度远快于激活值或有效低秩更新,且在探针适配器收敛前即可复用。为此提出ABM-LoRA:先快速训练标准适配器作为早期任务探针,再以其中预激活符号为层间目标,通过基于边距的铰链损失优化新适配器,二者均在后续微调前丢弃。该边界监督能揭示下游雅可比矩阵中被抑制或局部不可见的LoRA方向,真正迁移的是边界侧信息而非具体激活值或输出信号。因部分边界追踪已足够,ABM-LoRA仅需探针前向传播即可获得近似全训练收益,显著降低开销。在T5-base/GLUE、ConvNeXt-T和Swin-T细粒度分类、以及Qwen2.5-1.5B和LLaMA2-7B指令微调任务上优于标准LoRA,媲美或超越基于SVD和梯度的初始化器,且无需预处理。
原文摘要 · Abstract (English)
Low-Rank Adaptation (LoRA) is highly sensitive to initialization, yet existing schemes construct the initial subspace from statistics at the pretrained point, capturing pre-adaptation geometry rather than how the adapter must move during learning. We examine the early adaptation trajectory and uncover a temporal asymmetry: task-induced activation boundaries---the signs of layer-wise pre-activations---recover markedly faster than activation values or effective low-rank updates and become reusable well before the probe adapter converges. Motivated by this, we propose Activation Boundary Matching for LoRA (ABM-LoRA), which briefly trains a standard adapter as an early task probe, then uses its pre-activation signs as layer-wise targets for a fresh adapter under a margin-based hinge objective. Both are discarded before otherwise unchanged downstream fine-tuning. We show that this boundary-supervision lifting can expose LoRA directions attenuated or locally unobservable through the downstream Jacobian, and that what transfers is the boundary side---not the exact activation value or the output-level signal. Because a partial boundary trace suffices, ABM-LoRA recovers most of the benefit of a fully trained reference at a fraction of the overhead, requiring only probe forward passes. It improves over standard LoRA on T5-base/GLUE, ConvNeXt-T and Swin-T fine-grained classification, and instruction tuning with Qwen2.5-1.5B and LLaMA2-7B, and matches or surpasses SVD- and gradient-based initializers without their preprocessing.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。