用合成数据精准评估医疗模型在不同人群中的表现,提升AI可信度。
Enabling Granular Subgroup Level Model Evaluations by Generating Synthetic Medical Time Series
- 改进扩散模型,让合成数据更贴近真实分布
- 合成数据使小群体评估误差降低50%,72%~84%群体表现更优
- 无需真实病历即可实现隐私保护的细粒度模型评测
我们提出一种新框架,利用合成ICU时间序列数据不仅用于训练,还可严格、可信地评估预测模型,涵盖整体人群与细粒度人口亚群。基于先前的扩散与变分自编码器生成器(TimeDiff、HealthGen、TimeAutoDiff),我们引入Enhanced TimeAutoDiff,通过加入分布对齐惩罚项优化潜空间扩散目标。在MIMIC-III和eICU数据集上,针对24小时死亡率与住院时长二分类任务进行广泛测试。结果表明,Enhanced TimeAutoDiff将真实数据上合成数据的评估差距(TRTS gap)降低超过70%,实现Δ_{TRTS} ≤ 0.014 AUROC,同时保持训练有效性(Δ_{TSTR} ≈ 0.01)。对于32个交叉亚群,大容量合成数据集将亚群级AUROC估计误差减少高达50%,并在72%–84%的亚群中优于小规模真实测试集。本工作为重症医疗中可信、细粒度模型评估提供了可落地、隐私保护的路径,支持跨多样化患者群体的稳健性能分析,推动医学AI整体可信性。
原文摘要 · Abstract (English)
We present a novel framework for leveraging synthetic ICU time-series data not only to train but also to rigorously and trustworthily evaluate predictive models, both at the population level and within fine-grained demographic subgroups. Building on prior diffusion and VAE-based generators (TimeDiff, HealthGen, TimeAutoDiff), we introduce \textit{Enhanced TimeAutoDiff}, which augments the latent diffusion objective with distribution-alignment penalties. We extensively benchmark all models on MIMIC-III and eICU, on 24-hour mortality and binary length-of-stay tasks. Our results show that Enhanced TimeAutoDiff reduces the gap between real-on-synthetic and real-on-real evaluation (``TRTS gap'') by over 70\%, achieving $Δ_{TRTS} \leq 0.014$ AUROC, while preserving training utility ($Δ_{TSTR} \approx 0.01$). Crucially, for 32 intersectional subgroups, large synthetic cohorts cut subgroup-level AUROC estimation error by up to 50\% relative to small real test sets, and outperform them in 72--84\% of subgroups. This work provides a practical, privacy-preserving roadmap for trustworthy, granular model evaluation in critical care, enabling robust and reliable performance analysis across diverse patient populations without exposing sensitive EHR data, contributing to the overall trustworthiness of Medical AI.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。