用IVON优化LoRA，提升大模型精度与校准性，成本几乎不变。
Variational Low-Rank Adaptation Using IVON
- 用IVON替代AdamW进行变分微调，提升训练稳定性。
- Llama-2 7B上准确率提升2.8%，校准误差降低4.6%。
- 比其他贝叶斯方法更高效易实现，适合资源受限场景。
我们证明，变分学习可显著提升低秩适配（LoRA）的准确率与校准性，且成本增加有限。通过将AdamW替换为改进的变分在线牛顿（IVON）算法，对大型语言模型进行微调。在拥有70亿参数的Llama-2模型上，IVON相比AdamW使准确率提升2.8%，预期校准误差降低4.6%。其准确率优于其他贝叶斯方法，但计算开销更低，实现更简单。本工作进一步验证了IVON在大语言模型中的有效性。代码已公开于https://github.com/team-approx-bayes/ivon-lora。
原文摘要 · Abstract (English)
We show that variational learning can significantly improve the accuracy and calibration of Low-Rank Adaptation (LoRA) without a substantial increase in the cost. We replace AdamW by the Improved Variational Online Newton (IVON) algorithm to finetune large language models. For Llama-2 with 7 billion parameters, IVON improves the accuracy over AdamW by 2.8% and expected calibration error by 4.6%. The accuracy is also better than the other Bayesian alternatives, yet the cost is lower and the implementation is easier. Our work provides additional evidence for the effectiveness of IVON for large language models. The code is available at https://github.com/team-approx-bayes/ivon-lora.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。