针对数值越小越易缺失的数据,提出改进k-means聚类的新方法
MNAR-$k$-means: A $k$-means Clustering for Data Missing Not at Random with Magnitude-Decaying Probability
- 基于缺失值大小递减的约束设计聚类算法
- 显著降低聚类中心估计偏差,提升真实数据下的聚类效果
- 适合处理实际中常见的非随机缺失数据,如传感器或医疗记录
传统k-means聚类依赖完整特征计算距离,无法直接应用于含缺失值的数据。常规做法是仅使用观测值进行聚类,相当于用对应簇均值填补缺失值。然而,在缺失不随机(MNAR)情形下,由于缺失与数据值相关,此类均值填补会导致聚类中心估计失真,影响聚类性能。现实中MNAR机制普遍存在,亟需改进相关方法。本文聚焦于一种数值越小越易缺失的幅度衰减型MNAR场景,提出一种基于填补值大小约束的新k-means聚类方法,具有良好的数学可解释性。同时建立了该方法估计聚类中心的统计一致性,证明其收敛至完整数据的真实聚类中心,并通过交替优化算法求解目标函数。模拟实验验证了该方法在改善聚类结果和减少估计偏差方面的有效性;真实世界数据应用进一步展示了其实用价值。
原文摘要 · Abstract (English)
The classical $k$-means clustering, based on distances computed from all data features, cannot be directly applied to incomplete data with missing values. A natural extension of $k$-means to missing data is to involve only the observed positions in clustering, which is equivalent to imputing missing values by corresponding cluster means. However, for data missing not at random (MNAR), since missingness is related to data values, such a mean-imputation-based method may lead to the distortion of estimated cluster centers, resulting in a poor clustering result. Since MNAR mechanisms are very common in reality, it is necessary to improve the performance of $k$-means-based clustering methods for such data. In this paper, we focus on a magnitude-decaying MNAR scenario where data is more likely to be missing at positions with smaller absolute values, and we propose a novel $k$-means clustering method based on the constraint of the size of imputation values, which enjoys a good mathematical interpretation. Moreover, we establish the statistical consistency of the estimated cluster centers of the proposed method to the true cluster centers of fully observed data, and solve the optimization of the proposed loss function via an alternative minimization algorithm. Simulation experiments verify the effect of the proposed method in improving clustering results and reducing the bias of estimated cluster centers. Applications to real-world missing data further show the utility of the proposed method.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。