BaLoRA让大模型微调更可靠，同时提升精度并给出可信度评估。
BaLoRA: Bayesian Low-Rank Adaptation of Large Scale Models
- 用自适应噪声注入的贝叶斯方式扩展LoRA，参数和计算开销极小。
- 在语言与视觉任务中逼近全量微调精度，且不确定性估计更准确。
- 适合对可靠性要求高的场景，如材料科学中的零样本误差预测。
低秩微调（LoRA）已成为降低大模型微调成本的标准方法。然而，其基于点估计的低秩更新限制了表达能力，与全量微调仍存在显著性能差距，且缺乏内置不确定性量化，限制了在可靠性至关重要的场景中的应用。本文提出BaLoRA，一种新颖的贝叶斯版LoRA，通过输入自适应的贝叶斯参数化方式扩展LoRA矩阵，仅增加极少参数与计算开销。令人惊讶的是，该贝叶斯扩展不仅生成校准良好的不确定性估计，其底层的自适应噪声注入还显著提升了预测精度，在自然语言推理与视觉任务中大幅缩小了与全量微调的差距。应用于金属有机框架材料的带隙预测时，BaLoRA能实现零样本测试时的不确定性估计，其与模型误差的相关性优于训练好的LoRA集成模型，并随计算资源增加持续提升，且不牺牲准确性。
原文摘要 · Abstract (English)
Low-Rank Adaptation (LoRA) has become the standard for fine-tuning large pre-trained models at reduced computational cost. However, its low-rank point-estimate updates limit expressiveness, leave a persistent gap relative to full fine-tuning accuracy, and provide no built-in uncertainty quantification, limiting its applicability in settings where reliability matters as much as accuracy. We introduce BaLoRA, a Bayesian extension of LoRA with a novel input-adaptive Bayesian parameterization of LoRA matrices that adds minimal parameters and compute. Surprisingly, not only does the Bayesian extension yield well-calibrated uncertainty estimates, but the adaptive noise injection underlying our approach also significantly improves prediction accuracy, narrowing the gap with full fine-tuning across both natural language reasoning and vision tasks. When applied to band gap prediction in metal-organic frameworks, BaLoRA produces zero-shot test-time uncertainty estimates that correlate more strongly with model error than a trained ensemble of LoRA models, and improve monotonically with compute without sacrificing accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。