用虚拟病例模拟评估症状检查器对罕见病的诊断性能变化。
Evaluating Rare Disease Diagnostic Performance in Symptom Checkers: A Synthetic Vignette Simulation Approach
- 基于人类表型本体库生成虚拟病例，模拟算法更新前的诊断表现。
- 预测准确率提升与实际表现高度相关（召回率R²=0.83，精确率R²=0.78）。
- 适合罕见病诊断算法开发者用于低成本、可解释的预评估。
症状检查器（SCs）为用户提供个性化医疗信息，但其算法更新可能意外降低对罕见病的诊断性能，而现有方法难以在部署前有效评估。本文提出并验证了一种新型合成病例模拟方法，利用专家标注的罕见病表型知识库——人类表型本体库（HPO），生成合成病例，模拟用户访谈以预测算法更新对罕见病诊断性能的影响。通过回溯八次历史算法更新的实测数据验证，该方法对有表型频率信息的五种疾病，能准确预测召回率@8变化（R²=0.83，p=0.031）和精确率@8变化（R²=0.78，p=0.047）。该方法基于公开医学知识库，具备高透明度与可解释性，助力开发者低成本高效优化罕见病诊断能力。
原文摘要 · Abstract (English)
Symptom Checkers (SCs) provide medical information tailored to user symptoms. A critical challenge in SC development is preventing unexpected performance degradation for individual diseases, especially rare diseases, when updating algorithms. This risk stems from the lack of practical pre-deployment evaluation methods. For rare diseases, obtaining sufficient evaluation data from user feedback is difficult. To evaluate the impact of algorithm updates on the diagnostic performance for individual rare diseases before deployment, this study proposes and validates a novel Synthetic Vignette Simulation Approach. This approach aims to enable this essential evaluation efficiently and at a low cost. To estimate the impact of algorithm updates, we generated synthetic vignettes from disease-phenotype annotations in the Human Phenotype Ontology (HPO), a publicly available knowledge base for rare diseases curated by experts. Using these vignettes, we simulated SC interviews to predict changes in diagnostic performance. The effectiveness of this approach was validated retrospectively by comparing the predicted changes with actual performance metrics using the R-squared ($R^2$) coefficient. Our experiment, covering eight past algorithm updates for rare diseases, showed that the proposed method accurately predicted performance changes for diseases with phenotype frequency information in HPO (n=5). For these updates, we found a strong correlation for both Recall@8 change ($R^2$ = 0.83,$p$ = 0.031) and Precision@8 change ($R^2$ = 0.78,$p$ = 0.047). Our proposed method enables the pre-deployment evaluation of SC algorithm changes for individual rare diseases. This evaluation is based on a publicly available medical knowledge database created by experts, ensuring transparency and explainability for stakeholders. Additionally, SC developers can efficiently improve diagnostic performance at a low cost.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。