通过故意增加缺失数据来消除梯度偏差，提升不完整数据下的学习精度。
Increasing Missingness to Reduce Bias: Richardson-SGD with Missing Data
- 利用理查森外推法，通过生成更稀疏的缺失数据版本来抵消梯度偏差。
- 单步外推将偏差从O(‖p‖)降至O(‖p‖²)，显著改善优化效果。
- 适用于任意可计算梯度的参数模型，特别适合与MICE等方法结合使用。
随机梯度方法在大规模学习中至关重要，但处理不完整协变量时存在系统性偏差问题，尤其在参数化模型中，各种插补方法均引入梯度偏差。本文证明所有参数模型在不同插补下均呈现类似偏差，并精确刻画了其对缺失率向量$ p $的依赖关系，主导项为$ O(\|p\|) $。基于此分析，提出一种基于理查森外推的简单去偏方法：从已有不完整样本出发，主动生成更高缺失率的稀疏版本，结合两者的随机梯度以消除主要偏差项。理论证明，在多种缺失场景下，一步外推可使梯度偏差从$ O(\|p\|) $降至$ O(\|p\|^2) $。该方法计算高效、模型无关，适用于任何可通过插补计算随机梯度的参数损失函数。当缺失指示独立时，总体梯度偏差是$ p $的多重线性多项式，仅依赖于单个坐标缺失引起的误差；此时方法可推广为多步外推，逐次抵消高阶项。实验表明，理查森去偏在多种广义线性模型上提升了优化与估计性能，且能与MICE等常用插补方法正向协同。结果表明，反直觉地，对已有缺失数据进一步施加可控缺失，反而能提升不完整数据学习的准确性。
原文摘要 · Abstract (English)
Stochastic gradient methods are central to modern large-scale learning, but their use with incomplete covariates remains delicate since imputation schemes generally introduce systematic gradient biases, as shown for linear models. In this work, we prove that all parametric models exhibit similar gradient bias for various imputation procedures and characterize exactly the dependence on the missingness ratio vector $p$, with $O(\|p\|)$ as the leading term. We exploit this analysis to propose a simple debiasing procedure for stochastic gradient descent (SGD) with missing values based on Richardson extrapolation, which leverages the exact expression of the gradient bias. The key idea is to \emph{deliberately add missingness}: from an already incomplete observation, we generate a further-thinned version at a higher, controlled missingness level, and combine the two resulting stochastic gradients to cancel the leading bias term. We prove that one Richardson step reduces the gradient bias from $O(\|p\|)$ to $O(\|p\|^2)$ under several missingness scenarios. Our proposed method is computationally efficient, model-agnostic and applies to any parametric loss whose stochastic gradient can be computed after imputation. Furthermore, when missing indicators are independent, the population gradient bias is a multilinear polynomial in $p$ and depends only on population gradient errors induced by declaring a single coordinate missing. In this case, our method generalizes to a multi-step Richardson procedure which recursively cancels higher-order terms. Empirically, Richardson debiasing improves optimization and estimation across several generalized linear models and combines positively with widely used imputation procedures such as MICE. These results suggest that, somewhat counter-intuitively, adding controlled missingness on top of existing missing data can make stochastic learning from incomplete data more accurate.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。