提出消除小批量二次近似偏差的方法，提升深度学习优化与不确定性估计精度。
Debiasing Mini-Batch Quadratics for Applications in Deep Learning
- 分析小批量计算导致二次近似出现系统性偏差的机制
- 证明该偏差会影响二阶优化和拉普拉斯近似中的不确定性评估
- 设计并验证了有效的去偏策略，适用于实际深度学习场景
二次近似是机器学习方法的核心基础。例如，二阶优化器试图通过局部二次代理函数找到目标函数的牛顿步；网络损失函数的二阶近似还可用于通过拉普拉斯近似量化输出不确定性。然而，在深度学习中，由于全训练集计算不可行，相关量通常基于小批量计算，这会以复杂方式扭曲并引入随机二次近似的偏差，对应用产生不利影响。本文（i）表明该偏差引入系统误差，（ii）提供理论解释，（iii）阐明其在二阶优化和拉普拉斯近似中的重要性，（iv）提出并评估了去偏策略。
原文摘要 · Abstract (English)
Quadratic approximations form a fundamental building block of machine learning methods. E.g., second-order optimizers try to find the Newton step into the minimum of a local quadratic proxy to the objective function; and the second-order approximation of a network's loss function can be used to quantify the uncertainty of its outputs via the Laplace approximation. When computations on the entire training set are intractable - typical for deep learning - the relevant quantities are computed on mini-batches. This, however, distorts and biases the shape of the associated stochastic quadratic approximations in an intricate way with detrimental effects on applications. In this paper, we (i) show that this bias introduces a systematic error, (ii) provide a theoretical explanation for it, (iii) explain its relevance for second-order optimization and uncertainty quantification via the Laplace approximation in deep learning, and (iv) develop and evaluate debiasing strategies.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。