区分因果方向,揭示模型在病例分布变化下的表现差异
A causal viewpoint on prediction model performance under changes in case-mix: discrimination and calibration respond differently for prognosis and diagnosis predictions
- 按预测任务的因果方向划分,分析病例构成变化的影响
- 预后预测中校准稳定但区分度下降,诊断预测则相反
- 为临床模型部署提供基于因果结构的评估新视角
预测模型需在临床决策中保持可靠性能,用于诊断、预后和治疗规划。其性能通常通过区分度和校准度评估。数据分布变化会影响模型表现,尤其当当前应用与原评估场景的病例构成不同时。医疗中典型变化是病例构成偏移,如全科医生与三甲医院专科医生面对的患者群体不同。本文提出一种新框架,根据预测任务的因果方向,区分病例构成变化对区分度和校准度的不同影响:当预测处于因果方向(常见于预后预测)时,校准度保持稳定,但区分度下降;反之,在反因果方向(常见于诊断预测)时,区分度稳定而校准度恶化。通过模拟研究和心血管疾病预测模型的实证验证,展示了该框架的适用性。该因果病例构成框架有助于在不同临床场景中开发、评估和部署预测模型,强调理解预测任务因果结构的重要性。
原文摘要 · Abstract (English)
Prediction models need reliable predictive performance as they inform clinical decisions, aiding in diagnosis, prognosis, and treatment planning. The predictive performance of these models is typically assessed through discrimination and calibration. Changes in the distribution of the data impact model performance and there may be important changes between a model's current application and when and where its performance was last evaluated. In health-care, a typical change is a shift in case-mix. For example, for cardiovascular risk management, a general practitioner sees a different mix of patients than a specialist in a tertiary hospital. This work introduces a novel framework that differentiates the effects of case-mix shifts on discrimination and calibration based on the causal direction of the prediction task. When prediction is in the causal direction (often the case for prognosis predictions), calibration remains stable under case-mix shifts, while discrimination does not. Conversely, when predicting in the anti-causal direction (often with diagnosis predictions), discrimination remains stable, but calibration does not. A simulation study and empirical validation using cardiovascular disease prediction models demonstrate the implications of this framework. The causal case-mix framework provides insights for developing, evaluating and deploying prediction models across different clinical settings, emphasizing the importance of understanding the causal structure of the prediction task.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。