arXiv:2410.18144stat.MEcs.LG2024-10被引 11

纠正数据欠采样后模型预测偏差,提出更可靠的校准方法

Using Platt's scaling for calibration after undersampling -- limitations and how to address them

  • 用广义加性模型改进Platt校准,处理欠采样后的预测偏差
  • 模拟与案例研究显示传统Platt校准在欠采样后效果不佳
  • 新方法理论合理且跨场景表现稳定,适合高不平衡数据建模

当响应变量为二元且严重不平衡时,常采用基于响应的欠采样(保留多数类子集)以构建更平衡的训练集。然而,基于欠采样数据训练的模型(称为基础模型)生成的预测存在严重偏差。尽管Platt校准(通过逻辑回归建模基础模型预测值与真实响应的关系)常被用于校正此类偏差,但其原设计并非为此目的。本文首次系统研究了在欠采样后使用Platt校准的有效性。分析与仿真结果表明:若基础模型在全量数据上训练时无法被校准,则其在欠采样后也难以通过Platt校准修复。我们建议采用改进版校准方法——对基础模型预测的logit值拟合逻辑广义加性模型(GAM),该方法在理论和实证上均表现良好,适用于多种设置。

原文摘要 · Abstract (English)

When modelling data where the response is dichotomous and highly imbalanced, response-based sampling where a subset of the majority class is retained (i.e., undersampling) is often used to create more balanced training datasets prior to modelling. However, the models fit to this undersampled data, which we refer to as base models, generate predictions that are severely biased. There are several calibration methods that can be used to combat this bias, one of which is Platt's scaling. Here, a logistic regression model is used to model the relationship between the base model's original predictions and the response. Despite its popularity for calibrating models after undersampling, Platt's scaling was not designed for this purpose. Our work presents what we believe is the first detailed study focused on the validity of using Platt's scaling to calibrate models after undersampling. We show analytically, as well as via a simulation study and a case study, that Platt's scaling should not be used for calibration after undersampling without critical thought. If Platt's scaling would have been able to successfully calibrate the base model had it been trained on the entire dataset (i.e., without undersampling), then Platt's scaling might be appropriate for calibration after undersampling. If this is not the case, we recommend a modified version of Platt's scaling that fits a logistic generalized additive model to the logit of the base model's predictions, as it is both theoretically motivated and performed well across the settings considered in our study.

模型校准欠采样逻辑回归不平衡数据

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。