用博弈论方法自动找出表格数据中最关键的异常点。
Tab-Shapley: Identifying Top-k Tabular Data Quality Insights
- 基于合作博弈理论，用Shapley值衡量属性对异常的贡献。
- 在真实数据集上验证，能精准定位异常属性与记录组合。
- 无需标注，适合数据清洗和质量评估场景。
我们提出一种无监督方法，通过识别表格数据中前k个数据质量异常洞察来聚合异常。每个洞察包含一组异常属性及其对应的异常记录子集，作为用户可解释的证据。该过程面临三大挑战：(i) 缺乏标注异常；(ii) 子集搜索空间呈指数级增长；(iii) 属性间复杂依赖关系掩盖了真正的异常根源。传统频次方法无法捕捉这些依赖，导致结果不准确。为此，我们引入Tab-Shapley，一个基于合作博弈论的框架，利用Shapley值量化每个属性对数据异常性的贡献。尽管计算Shapley值通常需指数时间，我们证明该博弈具有闭式解，实现高效计算。我们在含真实异常标签的真实世界表格数据集上进行了实证分析，验证了该方法的有效性。
原文摘要 · Abstract (English)
We present an unsupervised method for aggregating anomalies in tabular datasets by identifying the top-k tabular data quality insights. Each insight consists of a set of anomalous attributes and the corresponding subsets of records that serve as evidence to the user. The process of identifying these insight blocks is challenging due to (i) the absence of labeled anomalies, (ii) the exponential size of the subset search space, and (iii) the complex dependencies among attributes, which obscure the true sources of anomalies. Simple frequency-based methods fail to capture these dependencies, leading to inaccurate results. To address this, we introduce Tab-Shapley, a cooperative game theory based framework that uses Shapley values to quantify the contribution of each attribute to the data's anomalous nature. While calculating Shapley values typically requires exponential time, we show that our game admits a closed-form solution, making the computation efficient. We validate the effectiveness of our approach through empirical analysis on real-world tabular datasets with ground-truth anomaly labels.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。