arXiv:2603.29824cs.LG2026-03被引 1

通过函数空间对齐,让低秩微调更快更准。

Curvature-Guided LoRA: Matching Full Fine-Tuning in Function Space

  • 在函数空间直接对齐输出,而非参数空间。
  • 利用局部曲率信息选择更新方向,提升收敛速度与性能。
  • 无需显式计算二阶矩阵,计算高效,适合大规模模型。

参数高效微调方法如LoRA可高效适配大型预训练模型,但通常在收敛速度和最终性能上仍落后于全量微调。现有方法试图通过使LoRA参数更新与全量微调对齐来缩小差距,但这种参数空间对齐仅间接影响模型输出。本文从函数空间视角出发,提出预测对齐问题,目标是使LoRA微调的输出与全量微调一致。我们证明该目标自然导向一种曲率感知的二阶公式,最优低秩更新对应一种类似牛顿法、曲率白化的梯度。基于此,我们提出曲率引导的LoRA(CG-LoRA),通过局部曲率信息选择适配方向。该方法计算高效,避免了显式构建二阶矩阵。在标准自然语言理解基准上的实验表明,相比现有LoRA变体,CG-LoRA在性能和收敛速度上均有提升。

原文摘要 · Abstract (English)

Parameter-efficient fine-tuning methods such as LoRA enable efficient adaptation of large pretrained models, but often lag behind full fine-tuning in both convergence speed and final performance. Recent approaches aim to reduce this gap by aligning LoRA parameter updates with those of full fine-tuning, but such parameter-space alignment only indirectly controls model predictions. Instead, we adopt a function-space perspective and formulate the \emph{prediction alignment problem}, whose objective is to match the outputs of LoRA fine-tuning to those of full fine-tuning. We show that this objective naturally leads to a curvature-aware, second-order formulation, where optimal low-rank updates correspond to a Newton-like, curvature-whitened gradient. Based on this insight, we propose Curvature-Guided LoRA (CG-LoRA), an algorithm that selects adaptation directions using local curvature information. Our method is computationally efficient and avoids explicit second-order matrix construction. Experiments on standard natural language understanding benchmarks demonstrate improved performance and faster convergence compared to existing LoRA variants.

LoRA微调曲率高效训练

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。