arXiv:2605.03724cs.LGcs.AI2026-05

LoRA微调的秩阈值可从12降为1,尤其在二分类任务中表现更优。

Rethinking the Rank Threshold for LoRA Fine-Tuning

论文配图:Rethinking the Rank Threshold for LoRA Fine-Tuning
图 1 · 摘自论文原文
  • 改用非对称的LoRA流形维数,降低模型容量要求
  • 交叉熵损失下无需秩阈值,理论保证更宽松
  • 二分类任务中秩为1时性能最优,适合实际应用

近期研究在神经切线核框架下提出,为避免虚假局部极小值,LoRA微调所需秩r需满足r(r+1)/2 > KN,建议在典型少样本RoBERTa设置中r ≥ 12。但该条件针对一般输出维度K,其紧性及在真实交叉熵损失下的实际意义尚不明确。本文给出三项结果:首先,将对称Sard计数替换为非对称的LoRA流形维数,得到更弱的容量要求 r(m+n) - r² > C*·KN,其中C* ≈ 1.35(高斯独立同分布特征下),在经典设置中r = 1即可满足;其次,在交叉熵损失下,Polyak–Łojasiewicz不等式使秩阈值完全消除;第三,基于Rademacher复杂度的界预测,当偏置项饱和时秩一具有最优方差,这恰好发生在二分类场景而不在K > 2时。实验表明,在四个GLUE风格的二分类任务、三种编码器架构及RoBERTa-large上,秩为1的性能与原有建议的r = 12相当;而在多分类MNLI任务中,最优秩高于1,符合预测。二分类的理论保证依赖标准NTK假设,多分类扩展留待未来工作。

原文摘要 · Abstract (English)

A recent landscape analysis of LoRA fine-tuning in the neural tangent kernel regime establishes a sufficient condition $r(r+1)/2 > KN$ on the LoRA rank $r$ for the absence of spurious local minima under squared-error loss, prescribing $r \geq 12$ on canonical few-shot RoBERTa setups. The condition is stated for general output dimension $K$, so its sharpness in any particular regime, and its practical implication for the cross-entropy loss actually used in fine-tuning, are open. We give three results that together reduce the prescribed rank to $r = 1$ for binary classification in this regime. First, replacing the symmetric Sard-form count with the non-symmetric LoRA manifold dimension yields a strictly weaker capacity requirement, $r(m+n) - r^2 > C^* \cdot KN$ with $C^* \approx 1.35$ under Gaussian-iid features, satisfied at $r = 1$ on canonical setups. Second, in the cross-entropy setting the Polyak--Łojasiewicz inequality removes the rank threshold entirely. Third, a Rademacher-complexity bound predicts rank-one variance optimality precisely when the bias term is saturated, which is the case for binary classification but not for $K > 2$. Empirically, across four GLUE-style binary tasks, three encoder architectures, and at scale on RoBERTa-large, rank one is competitive with the existing prescription $r = 12$; on multi-class MNLI the optimal rank shifts above one, also as predicted. The binary-regime guarantees are conditional on standard NTK assumptions; the multi-class extension is left to future work.

LoRA微调二分类理论分析

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。