提出一种无需二阶导数的在线协方差估计算法，提升SGD估计精度。
Refining Covariance Matrix Estimation in Stochastic Gradient Descent Through Bias Reduction
- 采用偏差减少技术，实现完全在线更新
- 收敛速度达 $n^{(α-1)/2} \sqrt{\log n}$，优于现有无黑塞方法
- 适合需要高精度协方差估计的在线学习场景
我们研究了随机梯度下降（SGD）算法的在线推断与渐近协方差估计问题。虽然已有插值法和批均值估计等经典方法，但它们或需不可得的二阶（黑塞）信息，或收敛缓慢。为此，我们提出一种全新的、完全在线的去偏协方差估计器，无需二阶导数即可显著提升估计精度。该方法通过偏差减少技术，实现 $n^{(α-1)/2} \sqrt{\log n}$ 的收敛速率，优于现有的无黑塞替代方案。
原文摘要 · Abstract (English)
We study online inference and asymptotic covariance estimation for the stochastic gradient descent (SGD) algorithm. While classical methods (such as plug-in and batch-means estimators) are available, they either require inaccessible second-order (Hessian) information or suffer from slow convergence. To address these challenges, we propose a novel, fully online de-biased covariance estimator that eliminates the need for second-order derivatives while significantly improving estimation accuracy. Our method employs a bias-reduction technique to achieve a convergence rate of $n^{(α-1)/2} \sqrt{\log n}$, outperforming existing Hessian-free alternatives.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。