让合成医疗数据更真实,同时保证实用性能达标。
Improving the Realism of Synthetic Clinical Benchmarks Under Utility Constraints
- 在不降低实用性的前提下,优化合成数据的结构真实性。
- 改进后缺失率降至40%以下,可操作行占比升至60%以上。
- 适合医疗AI开发者用于构建更可信的测试基准。
企业级AI代理的合成临床基准在通过现有实用性检验后,仍可能在隐私敏感的医疗环境中表现出结构性不真实。本文研究如何在不破坏现有实用性标准的前提下提升此类基准的真实性。将基准修订问题形式化为实用性约束下的真实性优化:数据修改需提升真实性,同时保持在操作实用性阈值之上。实验基于Synthea生成的患者数据,经过演示电子病历工作流处理,并使用与真实数据相同的下游分析管道。真实性通过缺失模式、简洁性、结构合理性及人群匹配度评估。基线基准极不真实:成对缺失率达79.44%,仅12.75%的记录可操作,38.94%患者无任何可操作指标,前三项词频集中度达100.0%。两种确定性修订方法在保持实用性的同时显著改善了真实性,而简单的数据填充控制仍保留不真实的模板特征。结果表明,内部真实性与源数据对整体操作参考的拟合度相关但独立,提示应显式优化合成基准质量,将实用性视为约束而非真实性的充分证明。
原文摘要 · Abstract (English)
Synthetic clinical benchmarks for enterprise AI agents can pass existing utility checks and still remain structurally unrealistic, especially in privacy-sensitive healthcare settings where operational data are hard to access. We study how to improve such benchmarks without breaking the downstream utility checks already used in practice. We formulate benchmark revision as utility-constrained realism improvement: dataset changes should increase realism while staying above an operational utility floor. We instantiate this idea on a care-gap benchmark derived from Synthea-generated patients exercised through demonstration electronic health record workflows and then processed by the same downstream pipeline as operational data. Realism is measured through missingness structure, simplicity, structural plausibility, and population alignment. The baseline benchmark is extremely thin: sampled-pair missingness is 79.44%, only 12.75% of rows are actionable, 38.94% of patients have zero actionable measures, and top-three token concentration reaches 100.0%. Two deterministic revisions improve these panels while remaining above the current utility floor, whereas a naive densification control preserves unrealistic templating. We further show that internal benchmark realism and source fidelity to an aggregate operational reference are related but distinct objectives. These results suggest that synthetic benchmark quality should be optimized explicitly, with utility treated as one constraint rather than as sufficient evidence of realism.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。