通过打破低秩适配的对称性，自动确定关键更新方向。
Learning Adapter Rank via Symmetry Breaking
- 用变分推断让低秩适配的坐标可辨识，选出最优方向基。
- 仅增 $/mathcal{O}(r)$ 参数，实现自适应秩与预测置信度联合学习。
- 比现有方法更紧凑，校准更稳定，训练成本相当。
低秩适配有效部分源于下游更新位于低维子空间，但LoRA的潜在秩坐标不可识别：任意可逆重参数化不改变权重更新。本文表明，采用对角秩向后验的变分推断可将这种非可识别性转化为有用的归纳偏置。通过打破LoRA的旋转规范对称性，变分目标在秩空间中选择了一个优选基，实现了对秩方向的重要性自动判定。由此提出低秩变分丢弃（LRVD），一种直接在低秩适配空间而非全参数空间进行推断的贝叶斯框架。作为实例，BayesLoRA 联合学习有效适配秩与预测不确定性，仅引入 $/mathcal{O}(r)$ 额外参数。实验显示，BayesLoRA 生成与学习更新主奇异方向一致的稳定秩结构，具备紧凑的预测校准能力，在相当训练成本下达到或超越强低秩稀疏化基线。
原文摘要 · Abstract (English)
Low-rank adaptation is effective partly because downstream updates lie in a low-dimensional subspace, but the latent rank coordinates of LoRA are not identifiable: any invertible reparameterization of the adapter factors leaves the weight update unchanged. We show that variational inference with a diagonal rank-wise posterior turns this non-identifiability into a useful inductive bias. By breaking LoRA's rotational gauge symmetry, the variational objective selects a preferred basis in rank space, enabling automatic relevance determination over rank directions. This yields Low-Rank Variational Dropout (LRVD), a Bayesian framework that performs inference directly in the low-rank adaptation space rather than the ambient weight space. As an instantiation, BayesLoRA jointly learns effective adapter rank and predictive uncertainty with only $\mathcal{O}(r)$ additional parameters. Empirically, BayesLoRA induces stable rank structure aligned with the dominant singular directions of learned updates, yields compact predictive calibration and matches or exceeds strong low-rank sparsification baselines at comparable training cost.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。