LoRA抗标签噪声有理论依据，能避免过拟合脏数据。
Why LoRA Resists Label Noise: A Theoretical Framework for Noise-Robust Parameter-Efficient Fine-Tuning
- 通过低秩分解限制模型记忆能力，防止过度拟合噪声标签。
- 样本量超阈值后，高噪声环境下误判率下降，泛化性更强。
- 适合在标注质量差的数据上微调大模型，如新闻分类任务。
参数高效微调方法如低秩适应（LoRA）已成为适配大型预训练模型的主流方式。本文提出一个理论框架，解释了LoRA一种未被充分研究的特性：对标签噪声的内在鲁棒性。分析揭示三个关键发现：首先，我们证明当样本量超过 $O(r(d+k-r))$ 时，秩为 $r$ 的LoRA无法记忆所有可能的标签分配，从而限制其拟合任意噪声的能力；其次，我们推导出最优秩平衡下的近似偏差与噪声引发的方差关系，显示其随噪声率上升而降低；第三，我们建立了时间分离机制：干净模式早期被学习，噪声记忆则滞后发生。为此，我们提出基于秩感知的课程训练（RACT），利用秩差异检测噪声。实验验证预测，RACT在AG News数据集上实现91.1%的噪声检测F1，同时保持91.46%准确率，优于无噪声检测能力的基线方法。
原文摘要 · Abstract (English)
Parameter-efficient fine-tuning methods like Low-Rank Adaptation (LoRA) have become the dominant paradigm for adapting large pretrained models. We present a theoretical framework explaining an underexplored property: LoRA's inherent resistance to label noise. Our analysis reveals three key insights. First, we prove that rank-$r$ LoRA cannot memorize all possible label assignments once the sample size exceeds $O(r(d+k-r))$, limiting its capacity to fit arbitrary noise. Second, we derive an optimal rank balancing approximation bias and noise-induced variance, showing it decreases with noise rate. Third, we establish temporal separation: clean patterns are learned early while noise memorization occurs later. We propose RACT (Rank-Aware Curriculum Training), leveraging rank discrepancy for noise detection. Experiments validate our predictions, with RACT achieving 91.1% F1 for noise detection on AG News while maintaining 91.46% accuracy, competitive with baselines that lack noise detection capability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。