用预训练矩阵提升神经网络权重不确定性估计效率
SOAP-Bubbles: Structured Weight Uncertainty for Neural Networks
- 在SOAP预处理空间中运行变分推断，生成非对角协方差
- 语言模型预训练中性能优于传统对角协方差方法
- 无需改动训练流程，适合大规模深度学习应用
结构化权重不确定性可提升深度学习多个方面，但其估计成本高且难实现。本文提出通过适配SOAP优化器解决该问题：将现有对角协方差变分方法IVON运行于SOAP预处理算子的特征空间，并利用该预处理算子将对角估计转换为非对角协方差。所得方法成本与SOAP相当，且无需修改训练流程。我们称此得到的后验分布为SOAP-Bubbles，新优化器为Eigenspace-VON（EVON）。实验表明，对于逻辑回归，EVON能恢复精确的高斯协方差；在语言模型预训练中，其效果显著优于现有对角协方差方法。本工作使大规模深度学习中更丰富的后验分布估计变得可行。
原文摘要 · Abstract (English)
Structured weight-uncertainty can improve many aspects of deep learning, but it remains costly to estimate and difficult to implement. Here, we show that these issues can be addressed by adapting the SOAP optimizer. Our key idea is to run IVON, an existing diagonal-covariance variational method, in the eigenspace of SOAP's preconditioner and then use the preconditioner to transform the diagonal estimate into a non-diagonal covariance. The resulting method has costs similar to those of SOAP and requires no drastic changes to training pipelines. We call the posteriors obtained in this way SOAP-Bubbles and our new optimizer Eigenspace-VON (EVON). We show that, for logistic regression, EVON recovers the exact Gaussian covariance and that, for language model pretraining, it yields significantly better results than existing diagonal-covariance methods. Our work makes it easier to estimate more expressive posterior distributions for deep learning at scale.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。