评测缺失值填补方法在真实社会场景下的表现,发现不同方法各有优劣。
Still More Shades of Null: An Evaluation Suite for Responsible Missing Value Imputation
- 构建多机制缺失与缺失迁移的真实场景评估框架
- 29,736次实验揭示填补方法在性能、公平性与稳定性间的权衡
- 适合关注数据伦理与实际应用的机器学习研究者
缺失值是科学界长期关注的实际问题。本文提出Shades-of-Null评估套件,用于负责任的缺失值填补评估。其创新点在于:(i) 建模超越经典MCAR、MAR、MNAR的现实社会相关缺失场景,包含多机制缺失(多种缺失模式共存)和缺失迁移(训练与测试阶段缺失机制变化);(ii) 全面评估填补质量、公平性,以及填补后模型的预测性能、公平性与稳定性。我们基于该套件开展大规模实证研究,涵盖29,736个实验流程,发现并无单一最优填补方法适用于所有缺失类型,但预测性能、公平性与稳定性之间存在显著权衡,取决于缺失场景、填补方法及下游模型架构的组合。我们公开发布Shades-of-Null,助力研究者在合理且具社会意义的场景中,多维度评估缺失值填补方法。
原文摘要 · Abstract (English)
Data missingness is a practical challenge of sustained interest to the scientific community. In this paper, we present Shades-of-Null, an evaluation suite for responsible missing value imputation. Our work is novel in two ways (i) we model realistic and socially-salient missingness scenarios that go beyond Rubin's classic Missing Completely at Random (MCAR), Missing At Random (MAR) and Missing Not At Random (MNAR) settings, to include multi-mechanism missingness (when different missingness patterns co-exist in the data) and missingness shift (when the missingness mechanism changes between training and test) (ii) we evaluate imputers holistically, based on imputation quality and imputation fairness, as well as on the predictive performance, fairness and stability of the models that are trained and tested on the data post-imputation. We use Shades-of-Null to conduct a large-scale empirical study involving 29,736 experimental pipelines, and find that while there is no single best-performing imputation approach for all missingness types, interesting trade-offs arise between predictive performance, fairness and stability, based on the combination of missingness scenario, imputer choice, and the architecture of the predictive model. We make Shades-of-Null publicly available, to enable researchers to rigorously evaluate missing value imputation methods on a wide range of metrics in plausible and socially meaningful scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。