arXiv:2410.15528cs.AI2024-10被引 8

Sporo AI医写助手比GPT-4o mini更准确,适合临床记录。

Improving Clinical Documentation with AI: A Comparative Study of Sporo AI Scribe and GPT-4o mini

  • 采用微调医疗大模型的多智能体系统,自动提取病患对话关键信息。
  • 在召回率、精确率和F1值上均优于GPT-4o mini,错误更少。
  • 医生评价其摘要更准确全面,适合临床工作使用。

AI医疗记录助手可缓解医疗文书负担。本研究对比了Sporo Health的AI医写系统与OpenAI的GPT-4o Mini在多项指标上的表现。基于去标识化患者对话数据集,将AI生成摘要与医师手写笔记(真实基准)进行对比,评估临床内容召回率、精确率及F1分数。同时通过修改版医师文档质量量表(PDQI-9)由医学生与医师进行评分。结果表明,Sporo AI在所有指标上均优于GPT-4o Mini,生成摘要更准确、完整且相关性更高,幻觉更少。研究证实Sporo AI Scribe是高效可靠的临床文书工具,能提升医生工作效率并保障隐私安全。

原文摘要 · Abstract (English)

AI-powered medical scribes have emerged as a promising solution to alleviate the documentation burden in healthcare. Ambient AI scribes provide real-time transcription and automated data entry into Electronic Health Records (EHRs), with the potential to improve efficiency, reduce costs, and enhance scalability. Despite early success, the accuracy of AI scribes remains critical, as errors can lead to significant clinical consequences. Additionally, AI scribes face challenges in handling the complexity and variability of medical language and ensuring the privacy of sensitive patient data. This case study aims to evaluate Sporo Health's AI scribe, a multi-agent system leveraging fine-tuned medical LLMs, by comparing its performance with OpenAI's GPT-4o Mini on multiple performance metrics. Using a dataset of de-identified patient conversation transcripts, AI-generated summaries were compared to clinician-generated notes (the ground truth) based on clinical content recall, precision, and F1 scores. Evaluations were further supplemented by clinician satisfaction assessments using a modified Physician Documentation Quality Instrument revision 9 (PDQI-9), rated by both a medical student and a physician. The results show that Sporo AI consistently outperformed GPT-4o Mini, achieving higher recall, precision, and overall F1 scores. Moreover, the AI generated summaries provided by Sporo were rated more favorably in terms of accuracy, comprehensiveness, and relevance, with fewer hallucinations. These findings demonstrate that Sporo AI Scribe is an effective and reliable tool for clinical documentation, enhancing clinician workflows while maintaining high standards of privacy and security.

AI医写临床记录大模型应用

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。