用极低参数量实现大模型不确定性量化，既准又快。
Minimal Ranks, Maximum Confidence: Parameter-efficient Uncertainty Quantification for LoRA
- 通过子空间推断在极低维参数空间建模不确定性
- 仅用少量参数就实现更好校准与泛化性能
- 适合追求高效且可信推理的AI研发人员
低秩适配（LoRA）通过将权重更新分解为低秩矩阵，实现了大语言模型的参数高效微调，显著降低存储与计算开销。然而，标准LoRA缺乏不确定性量化机制，导致模型过于自信且校准不佳。贝叶斯变体虽可解决此问题，但大幅增加可训练参数，削弱了原有效率优势，且训练更困难、收敛不稳定。本文提出一种新型参数高效的贝叶斯LoRA方法——基于子空间推断，证明在极低维参数空间中仍可有效建模不确定性。实验表明：（1）在合适的权重空间投影下，不确定性可在低维空间中有效建模；（2）权重协方差具有低秩特性。该方法在保持计算高效的同时，显著提升模型校准与泛化能力。
原文摘要 · Abstract (English)
Low-Rank Adaptation (LoRA) enables parameter-efficient fine-tuning of large language models by decomposing weight updates into low-rank matrices, significantly reducing storage and computational overhead. While effective, standard LoRA lacks mechanisms for uncertainty quantification, leading to overconfident and poorly calibrated models. Bayesian variants of LoRA address this limitation, but at the cost of a significantly increased number of trainable parameters, partially offsetting the original efficiency gains. Additionally, these models are harder to train and may suffer from unstable convergence. In this work, we propose a novel parameter-efficient Bayesian LoRA via subspace inference, demonstrating that effective uncertainty quantification can be achieved in very low-dimensional parameter spaces. The proposed method achieves strong performance with improved calibration and generalization while maintaining computational efficiency. Our empirical findings show that, with the appropriate projection of the weight space: (1) uncertainty can be effectively modeled in a low-dimensional space, and (2) weight covariances exhibit low ranks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。