改进大模型微调的低秩适配，提升收敛速度与性能
PoLAR: Polar-Decomposed Low-Rank Adapter Representation
- 用极分解思想拆分低秩更新，分离方向与尺度
- 理论证明收敛速度呈指数级提升，实测在3个基准上均增效
- 适合大模型微调场景，尤其对小模型更显著
我们发现，大规模模型的低秩适配存在稳定秩过低的问题，远低于子空间的线性代数秩，导致微调性能下降。为缓解子空间利用不足，我们提出 PoLAR，一种受极分解启发的参数化方法，将低秩更新分解为两个约束在 Stiefel 流形上的方向矩阵和一个无约束的尺度矩阵。理论表明，PoLAR 在典型低秩适配任务中可实现指数级更快的收敛速度。结合黎曼优化，在三个不同基准上（涵盖通用语言理解、常识推理、数学求解）均取得一致提升，基座模型规模从 350M 到 27B 不等。
原文摘要 · Abstract (English)
We show that low-rank adaptation of large-scale models suffers from a low stable rank that is well below the linear algebraic rank of the subspace, degrading fine-tuning performance. To mitigate the underutilization of the allocated subspace, we propose PoLAR, a parameterization inspired by the polar decomposition that factorizes the low-rank update into two direction matrices constrained to Stiefel manifolds and an unconstrained scale matrix. Our theory shows that PoLAR yields an exponentially faster convergence rate on a canonical low-rank adaptation problem. Pairing the parameterization with Riemannian optimization leads to consistent gains on three different benchmarks testing general language understanding, commonsense reasoning, and mathematical problem solving with base model sizes ranging from 350M to 27B.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。