提出机制检测法，识别数据错误是否由数据本身引发
MechDetect: Detecting Data-Dependent Errors
- 基于机器学习分析错误是否依赖于数据内容
- 在基准数据集上验证了错误机制识别的有效性
- 适用于有错误标记的各类数据错误检测
数据质量监控是现代信息处理系统的核心挑战。尽管已有多种检测数据错误或数据漂移的方法，但很少有研究关注错误生成的机制。我们认为，了解错误是如何生成的，对于追踪和修复错误至关重要。本文在统计学文献关于缺失值的研究基础上，提出 MechDetect——一种简单算法，用于探究错误生成机制。给定一个表格数据集和相应的错误掩码，该算法利用机器学习模型判断错误是否依赖于数据本身。本工作扩展了现有缺失值机制检测方法，可直接应用于其他类型的错误，前提是存在错误掩码。我们在多个基准数据集上通过实验验证了 MechDetect 的有效性。
原文摘要 · Abstract (English)
Data quality monitoring is a core challenge in modern information processing systems. While many approaches to detect data errors or shifts have been proposed, few studies investigate the mechanisms governing error generation. We argue that knowing how errors were generated can be key to tracing and fixing them. In this study, we build on existing work in the statistics literature on missing values and propose MechDetect, a simple algorithm to investigate error generation mechanisms. Given a tabular data set and a corresponding error mask, the algorithm estimates whether or not the errors depend on the data using machine learning models. Our work extends established approaches to detect mechanisms underlying missing values and can be readily applied to other error types, provided that an error mask is available. We demonstrate the effectiveness of MechDetect in experiments on established benchmark datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。