揭示LoRA微调的最优秩选择原则，解释为何过拟合会伤害性能。
Tight Sample Complexity for Low-Rank Adaptation: Matching Bounds and Rank Selection
- 通过局部Rademacher分析建立上界O~(rd/n)，证明低秩适配的有效性。
- 构造Fano型打包证明下界Omega(rd/n)，首次实现理论匹配。
- 发现不加正则的优化器过拟合有害，而正则化方法可容忍过参数化。
低秩适配（LoRA）已成为微调大模型的标准方法，但其统计性质仍不完全清晰。现有泛化结果仅给出O~(sqrt(rd/n))或O~(rd/n)的上界，缺乏匹配的下界，且如何选择LoRA秩r尚无理论依据。本文通过局部Rademacher论证，建立了在目标适配秩不超过r时，经验风险最小化器的超额风险上界为O~(rd/n)。进一步通过Fano型秩-r子空间的打包构造，证明了任意输出在秩-r LoRA类中的估计器均存在Omega(rd/n)的极小极大下界。两者结合得出秩选择二分法：对受限经验风险最小化器，最优秩等于内在秩r*，过拟合会恶化性能；对核范数-截断类自适应估计器，过拟合无害，速率恒为Theta~(r* d / n)，与所选秩无关。三者共同刻画了在良好设定下的局部二次框架中LoRA微调的统计复杂度，并将观察到的过参数化惩罚归因于未正则化的优化器，而非LoRA结构本身。理论预测在合成迹回归基准和三个真实场景（DistilBERT/RoBERTa + SST-2/MRPC）中得到验证，所有配置均呈现验证损失的U形曲线，其中两个显示大秩下损失显著上升（配对置换检验p=0.016）。
原文摘要 · Abstract (English)
Low-Rank Adaptation (LoRA) has become the standard mechanism for fine-tuning large pretrained models, yet its statistical properties remain only partially understood. Existing generalization results provide upper bounds of the form O~(sqrt(rd/n)) or O~(rd/n), but a matching lower bound is missing, and the question of how to choose the LoRA rank r has no formal answer. Both gaps are closed here. A local Rademacher argument establishes an upper bound of O~(rd/n) on the excess risk of the empirical risk minimizer over rank-r LoRA, whenever the target adaptation has rank at most r. A matching minimax lower bound of Omega(rd/n) is then proved via a Fano-type packing of the rank-r subspace of R^{d x d}; the bound applies to any estimator whose output lies in the rank-r LoRA class. Combining the two yields a rank-selection dichotomy. For the constrained empirical risk minimizer, the optimal rank equals the intrinsic rank r*, and over-ranking strictly hurts. For adaptive estimators of the nuclear-norm-then-truncate type, over-ranking is harmless and the rate saturates at Theta~(r* d / n) regardless of r. Taken together, the three results characterize the statistical complexity of LoRA fine-tuning within the well-specified locally quadratic regime, and identify the empirically observed over-parameterization penalty as a property of unregularized empirical risk minimization rather than of the LoRA class itself. Predictions of the theory are verified on a synthetic trace-regression benchmark and on real LoRA fine-tuning across three (model, task) configurations covering DistilBERT and RoBERTa on SST-2 and MRPC. All configurations exhibit the predicted U-shape in validation loss, with two showing statistically significant loss inflation at large ranks (paired permutation p = 0.016).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。