在低维空间中实现高效贝叶斯微调，兼顾不确定性量化与计算效率。
Bayesian Fine-tuning in Projected Subspaces
- 通过投影权重空间，在极低维参数空间建模不确定性
- 保持低秩微调效率，同时显著提升模型校准能力
- 适合需要可信置信度的高可靠性场景
低秩适配（LoRA）通过将权重更新分解为低秩矩阵，实现了大模型的参数高效微调，大幅降低存储与计算开销。然而，标准LoRA缺乏不确定性量化机制，导致模型过于自信且校准不佳。虽有贝叶斯版LoRA解决此问题，但训练参数显著增加，部分抵消了原始效率优势，且训练更困难、收敛不稳定。本文提出一种新型参数高效的贝叶斯微调框架，证明在极低维参数空间中即可实现有效的不确定性建模。实验表明，适当的权重空间投影下，不确定性可在低维空间有效表示，且权重协方差具有低秩特性，从而在保持计算高效的同时，获得更强性能、更好校准与泛化能力。
原文摘要 · Abstract (English)
Low-Rank Adaptation (LoRA) enables parameter-efficient fine-tuning of large models by decomposing weight updates into low-rank matrices, significantly reducing storage and computational overhead. While effective, standard LoRA lacks mechanisms for uncertainty quantification, leading to overconfident and poorly calibrated models. Bayesian variants of LoRA address this limitation, but at the cost of a significantly increased number of trainable parameters, partially offsetting the original efficiency gains. Additionally, these models are harder to train and may suffer from unstable convergence. In this work, we propose a novel framework for parameter-efficient Bayesian fine-tuning, demonstrating that effective uncertainty quantification can be achieved in very low-dimensional parameter spaces. The proposed method achieves strong performance with improved calibration and generalization while maintaining computational efficiency. Our empirical findings show that, with the appropriate projection of the weight space uncertainty can be effectively modeled in a low-dimensional space, and weight covariances exhibit low ranks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。