用特征重要性集中度预判模型在分布偏移下的预测失效风险
Diagnosing Conformal Prediction Failures Under Distribution Shift: A COVID-19 Case Study
- 通过特征重要性集中度诊断梯度提升模型的预测失效脆弱性
- 同一数据集上覆盖概率下降幅度从轻微到灾难性不等,集中度越高越严重
- 适合关注模型部署可靠性的工程师,尤其针对梯度提升类模型
分位数校准预测在分布偏移下性能会退化,但从业者缺乏部署前预测失效的能力。本文提出使用SHAP重要性集中度——即最高特征的重要性占比——作为梯度提升分类器在分布偏移下对校准预测脆弱性的预诊断工具。以新冠疫情期间的供应链任务为真实案例,8个任务经历相同的时间偏移,但覆盖率下降程度从微小到灾难性不等。在9个领域共16个多分类任务中,特征重要性集中度与失效严重性呈强相关。标准分布偏移检测器虽能统一识别偏移,却无法区分严重与稳健结果。跨9个非供应链数据集的外部验证显示部分可迁移性。理论证明,在特定假设下,自适应预测集的置信得分边界随集中度单调恶化,且经实证验证。该诊断可识别梯度提升模型特有的依赖集中型失败,但无法检测神经网络中的全局敏感型失败。决策框架将诊断转化为带不确定性带的预部署探索规则。
原文摘要 · Abstract (English)
Conformal prediction provides distribution-free coverage guarantees, but these degrade under distribution shift - and practitioners lack tools to anticipate which deployed models will fail before observing test data. We propose SHapley Additive exPlanations (SHAP) concentration - the fraction of feature importance concentrated in the top feature - as a pre-deployment diagnostic for conformal prediction vulnerability in gradient-boosted classifiers. Using COVID-19 as a naturalistic case study, eight supply chain tasks experience identical temporal shift yet coverage drops ranging from negligible to catastrophic. Feature-importance concentration is strongly associated with failure severity across 16 multiclass tasks in 9 domains, while standard distributional shift detectors detect shift uniformly across tasks but cannot distinguish catastrophic from robust outcomes. External validation across 9 non-supply-chain datasets shows partial transfer. We prove a formal theorem showing that Adaptive Prediction Sets conformity-score bounds worsen monotonically with concentration under explicit assumptions, verified empirically. The diagnostic identifies concentrated-dependence failures characteristic of gradient-boosted models but does not detect global-sensitivity failures observed in neural networks. A decision framework operationalizes the diagnostic as an exploratory pre-deployment rule with an uncertainty band around a concentration threshold.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。