让大模型更敢说‘不知道’，提升推理可靠性
Bayesian-LoRA: Probabilistic Low-Rank Adaptation of Large Language Models
- 用概率化低秩适配替代传统确定性微调，引入不确定性感知
- 在30B模型上实现84%误差校准改善，训练成本仅增1.2倍
- 适合需要可靠置信度评估的医疗、金融等高风险场景
大型语言模型通常更关注准确率，即使不确定也会做出预测，尤其在小数据集微调时校准性严重不足。本文提出贝叶斯LoRA，将确定性的LoRA更新重构为受稀疏高斯过程启发的概率低秩表示。我们发现LoRA分解与克罗内克分解的后验分布存在结构同构，当后验不确定性坍缩时，LoRA即为其极限情况。我们在多种LLM架构和常识推理基准上进行了广泛实验。仅增加约0.42M参数，训练成本约为标准LoRA的1.2倍，贝叶斯LoRA在高达300亿参数的模型上显著改善校准性能，实现最高84%的ECE降低和76%的NLL降低，同时在分布内与分布外评估中保持竞争力的准确性。
原文摘要 · Abstract (English)
Large Language Models usually put more emphasis on accuracy and therefore, will guess even when not certain about the prediction, which is especially severe when fine-tuned on small datasets due to the inherent tendency toward miscalibration. In this work, we introduce Bayesian-LoRA, which reformulates the deterministic LoRA update as a probabilistic low-rank representation inspired by Sparse Gaussian Processes. We identify a structural isomorphism between LoRA's factorization and Kronecker-factored SGP posteriors, and show that LoRA emerges as a limiting case when posterior uncertainty collapses. We conduct extensive experiments on various LLM architectures across commonsense reasoning benchmarks. With only approximately 0.42M additional parameters and ${\approx}1.2{\times}$ training cost relative to standard LoRA, Bayesian-LoRA significantly improves calibration across models up to 30B, achieving up to 84% ECE reduction and 76% NLL reduction while maintaining competitive accuracy for both in-distribution and out-of-distribution (OoD) evaluations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。