arXiv:2606.29184cs.LG2026-06

动态分配低秩适配能力,让模型在小数据下更准确、不确定性更可信。

BaRA: Bayesian Adaptive Rank Allocation for Parameter-Efficient Fine-Tuning

  • 基于概率主题模型思想,按上下文动态激活稀疏的潜在因子。
  • 在多个语言任务上超越标准LoRA和现有贝叶斯方法,提升精度与校准度。
  • 适合追求高效微调且重视不确定性的研究者或实际应用者。

低秩适配(LoRA)通过固定低秩子空间约束参数更新,实现高效微调,但其刚性设计限制了表达灵活性,常导致预测过度自信和不确定性校准不足,尤其在数据稀缺时。现有贝叶斯LoRA虽改进了不确定性估计,但多依赖固定或启发式设定的秩,忽略了适配容量的上下文依赖性。本文提出贝叶斯自适应秩分配框架BaRA,受概率主题模型启发,通过激活稀疏、上下文相关的解耦潜在因子,实现实例级有效秩的动态变化。该贝叶斯建模提供原则性、数据驱动的容量控制,缓解过参数化问题,同时保持表达力。理论分析表明,BaRA的泛化误差取决于全局-局部门控机制诱导的联合有效秩 $ar{s}_{Φ,θ}$,而非最大秩 $r$,解释了为何稀疏自适应秩分配可降低有效假设复杂度而不牺牲输入相关表达力。在多种自然语言基准上的实验显示,相比标准LoRA及现有贝叶斯变体,BaRA在预测性能、鲁棒性与不确定性校准方面均有持续提升。

原文摘要 · Abstract (English)

While Low-rank adaptation (LoRA) enables highly efficient fine-tuning by constraining task-specific updates to fixed low-rank subspaces, this rigid design limits representational flexibility and often results in overconfident predictions and miscalibrated uncertainty, especially in low-data regimes. Recent Bayesian LoRA variants improve uncertainty estimation by modeling posterior distributions over adaptation parameters. However, these approaches typically rely on fixed or heuristically determined ranks, overlooking the inherently context-dependent nature of adaptation capacity. In this paper, we propose BaRA, a Bayesian Adaptive Rank Allocation framework for parameter-efficient fine-tuning. Drawing inspiration from probabilistic topic models, BaRA dynamically allocates adaptation capacity by activating a sparse, context-dependent subset of disentangled latent factors, enabling instance-wise variation in effective rank. This Bayesian formulation provides principled, data-driven capacity control, mitigating over-parameterization while preserving expressiveness. Beyond the modeling contribution, we provide a complexity-theoretic generalization analysis showing that the generalization gap of BaRA depends on the learned joint effective rank $\bar{s}_{Φ,θ}$ induced by the global-local gate, rather than the maximum rank $r$. This result explains why sparse adaptive rank allocation can reduce the effective hypothesis complexity while preserving input-dependent expressiveness. Extensive experiments on diverse natural language benchmarks demonstrate that BaRA consistently improves predictive performance, robustness, and uncertainty calibration compared to standard LoRA and existing Bayesian LoRA variants.

低秩微调贝叶斯方法不确定性校准

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。