通过特征子空间定位,可解释地检测数据分布偏移。
Unsupervised Domain Shift Detection with Interpretable Subspace Attribution

- 基于高维特征空间密度异常检测局部偏移
- 在20维基准和782维心电图数据中成功识别偏移
- 适合发现隐性群体偏差,用于模型前预处理
我们开发了一种检测领域偏移的工具,即数据集概率分布中的细微差异。通过算法识别高维特征空间中的局部密度异常,若发现异常,则进一步定位其最显著的特征子空间,使偏移可解释。此外,提出一种协议,从两个无标签数据集中提取无显著分布差异的样本子集以补偿偏移。在具有已知真实情况的20维受控基准上验证了该框架,成功恢复了整体与局部偏移及其对应的特征子空间。随后应用于782维健康心电图数据,在年龄与性别匹配但设备构成不同的队列间,检测到设备引起的偏移,提取出含不平衡设备成分的代表性子集,并识别出与采集差异相关的ECG特征。结果表明,密度偏移检测与子空间归因提供了一种实用框架,可在下游建模前揭示隐藏的群体偏差。
原文摘要 · Abstract (English)
We developed a tool for detecting domain shifts, namely subtle differences in the probability distributions of datasets. We identify these shifts using an algorithm designed to detect localised density anomalies in high-dimensional feature spaces. If an anomaly is present, we then identify the feature subspace in which the anomaly is most pronounced. This allows us to trace the domain shift to a small set of features, making the shift interpretable. Moreover, we provide a protocol for compensating domain shifts by extracting, from two unlabelled datasets, subsets of samples with no detectable residual distributional difference. We validate the framework on controlled 20-dimensional benchmarks with known ground truth, recovering both broad and localized shifts together with their supporting feature subspaces. We then apply it to healthy electrocardiogram (ECG) recordings represented by 782 features. In age- and sex-matched cohort comparisons differing in measurement-device composition, the method detects device-induced shifts, extracts representative subsets enriched in the imbalanced device components, and identifies ECG features associated with the acquisition contrast. These results suggest that density-shift detection and subspace attribution provide a practical framework for uncovering hidden cohort biases before downstream modelling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。