提出可控制奇异值范围的低秩微调方法，提升模型稳定性和性能。
Norm-Bounded Low-Rank Adaptation
- 通过显式约束奇异值范围实现低秩适配参数化
- 在语言和视觉任务中表现优于或等同于现有方法
- 对学习率、训练轮数等超参数更鲁棒，适合实际部署
本文提出一种新的参数高效微调方法——范数有界低秩适配（NB-LoRA）。该方法对低秩权重适配进行新型参数化，能够对适配矩阵的每个奇异值施加明确边界，从而满足任意酉不变范数约束，包括核范数、弗罗贝尼乌斯范数和谱范数。所提参数化为无约束、光滑且完备的，覆盖所有满足指定秩和奇异值约束的矩阵。自然语言生成实验表明，NB-LoRA 在性能上达到或超越现有 LoRA 方法，同时表现出更强的超参数鲁棒性。视觉微调实验显示，NB-LoRA 能有效避免模型灾难性遗忘，且在适应性能上仅带来微小损失；相比现有方法，其对适配秩、学习率和训练轮数等超参数更加鲁棒。
原文摘要 · Abstract (English)
In this work, we propose norm-bounded low-rank adaptation (NB-LoRA) for parameter-efficient fine tuning. NB-LoRA is a novel parameterization of low-rank weight adaptations that admits explicit bounds on each singular value of the adaptation matrix, which can thereby satisfy any prescribed unitarily invariant norm bound, including the Schatten norms (e.g., nuclear, Frobenius, spectral norm). The proposed parameterization is unconstrained, smooth, and complete, i.e. it covers all matrices satisfying the prescribed rank and singular-value bounds. Natural language generation experiments show that NB-LoRA matches or surpasses performance of competing LoRA methods, while exhibiting stronger hyper-parameter robustness. Vision fine-tuning experiments show that NB-LoRA can avoid model catastrophic forgetting without minor cost on adaptation performance, and compared to existing approaches it is substantially more robust to a hyper-parameters such as including adaptation rank, learning rate and number of training epochs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。