用贝叶斯视角改进参数高效微调，提升效率与性能
A Bayesian Interpretation of Adaptive Low-Rank Adaptation
- 引入信噪比与IVON优化器实现更可靠的参数重要性评估
- 在保持性能的同时，速度优于使用Adam的AdaLoRA
- 揭示参数重要性主要由幅度决定，而非方差
受自适应低秩适配（AdaLoRA）中基于敏感度的重要性评分启发，我们采用更具理论支持的指标，包括信噪比（SNR），并结合改进的变分在线牛顿（IVON）优化器，实现自适应参数预算分配。所得贝叶斯版本不仅性能达到或超过基于敏感度的指标，且相比使用Adam的AdaLoRA更快。理论分析揭示了两种度量之间的显著关联，为敏感度作为重要性评分的有效性提供了贝叶斯解释。此外，研究发现参数的重要性主要由其幅度决定，而非方差。
原文摘要 · Abstract (English)
Motivated by the sensitivity-based importance score of the adaptive low-rank adaptation (AdaLoRA), we utilize more theoretically supported metrics, including the signal-to-noise ratio (SNR), along with the Improved Variational Online Newton (IVON) optimizer, for adaptive parameter budget allocation. The resulting Bayesian counterpart not only has matched or surpassed the performance of using the sensitivity-based importance metric but is also a faster alternative to AdaLoRA with Adam. Our theoretical analysis reveals a significant connection between the two metrics, providing a Bayesian perspective on the efficacy of sensitivity as an importance score. Furthermore, our findings suggest that the magnitude, rather than the variance, is the primary indicator of the importance of parameters.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。