揭示高维不平衡分类中过拟合的统计机制,解释少数类为何更易受损。
A statistical theory of overfitting for imbalanced classification
- 基于高维渐近分析,发现逻辑回归与SVM的对数几率分布会畸变
- 少数类在训练集上对数几率呈截断正态分布,导致过拟合更严重
- 提出边界重平衡策略,适合处理医疗、欺诈等少数类重要场景
不平衡分类在数据分析中常见,少数类样本占比远低于多数类。经典大样本统计理论在高维数据下失效,无法解释实际机器学习中的过拟合现象。本文通过研究支持向量机和逻辑回归,建立高维不平衡分类的统计理论。发现维度升高会导致对数几率分布发生截断或偏斜,其性质可通过高维渐近下的变分问题刻画。对于两组分高斯混合生成的线性可分数据,测试集上的对数几率服从标准正态分布 $\mathsf{N}(0,1)$,而训练集上则渐近服从截断正态分布 $\max\{κ, \mathsf{N}(0,1)\}$,该现象已在表格、图像和文本数据中验证。该机制解释了少数类更易受过拟合影响的原因。进一步证明,在损失函数中引入类别大小进行边界重平衡,对缓解少数类准确率下降至关重要。理论还揭示了过拟合对校准性和不确定性量化的影响。
原文摘要 · Abstract (English)
Classification with imbalanced data is a common challenge in data analysis, where certain classes (minority classes) account for a small fraction of the training data compared with other classes (majority classes). Classical statistical theory based on large-sample asymptotics and finite-sample corrections is often ineffective for high-dimensional data, leaving many overfitting phenomena in empirical machine learning unexplained. In this paper, we develop a statistical theory for high-dimensional imbalanced classification by investigating support vector machines and logistic regression. We find that dimensionality induces truncation or skewing effects on the logit distribution, which we characterize via a variational problem under high-dimensional asymptotics. In particular, for linearly separable data generated from a two-component Gaussian mixture model, the logits from each class follow a normal distribution $\mathsf{N}(0,1)$ on the testing set, but asymptotically follow a rectified normal distribution $\max\{κ, \mathsf{N}(0,1)\}$ on the training set -- which is a pervasive phenomenon we verified on tabular data, image data, and text data. This phenomenon explains why the minority class is more severely affected by overfitting. Further, we show that margin rebalancing, which incorporates class sizes into the loss function, is crucial for mitigating the accuracy drop for the minority class. Our theory also provides insights into the effects of overfitting on calibration and other uncertain quantification measures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。