XMedGPT让医学AI能解释图像与文本,提升医生信任度。
Multi-Modal Explainable Medical AI Assistant for Trustworthy Human-AI Collaboration
- 融合图文解释,定位病灶区域增强可读性
- 图像标注准确率IoU达0.703,预测生存期优于前代26.9%
- 支持医生互动问答,量化不确定性能指导临床决策
通用医学AI(GMAI)在生物医学感知任务中已达到专家水平,但因多模态解释能力不足和预后能力有限,临床应用受限。本文提出面向医生的XMedGPT多模态AI助手,整合文本与视觉可解释性,实现透明可信的医疗决策支持。该模型不仅输出精准诊断结果,还能在医学影像中定位相关解剖部位,填补解释空白。为支持实际部署,引入基于交互问答的一致性评估不确定性指数。在四大维度验证:多模态可解释性、不确定性量化、预后建模及严格基准测试。模型在141个解剖区域上达到0.703的IoU,Kendall's tau-b为0.479,表明视觉推理与临床结果高度一致。在视觉问答任务中不确定性估计AUC达0.862,在放射报告生成中达0.764。针对肺癌与胶质瘤的生存与复发预测,相比现有领先模型提升26.9%,优于GPT-4o 25.0%。跨347个数据集、40种成像模态的基准测试及4个解剖系统的外部验证显示卓越泛化性,域内评估性能超越现有GMAI 20.7%,11,530条院内数据评估提升16.7%。整体推动以医生为中心的可信可扩展医疗AI发展。
原文摘要 · Abstract (English)
Generalist Medical AI (GMAI) systems have demonstrated expert-level performance in biomedical perception tasks, yet their clinical utility remains limited by inadequate multi-modal explainability and suboptimal prognostic capabilities. Here, we present XMedGPT, a clinician-centric, multi-modal AI assistant that integrates textual and visual interpretability to support transparent and trustworthy medical decision-making. XMedGPT not only produces accurate diagnostic and descriptive outputs, but also grounds referenced anatomical sites within medical images, bridging critical gaps in interpretability and enhancing clinician usability. To support real-world deployment, we introduce a reliability indexing mechanism that quantifies uncertainty through consistency-based assessment via interactive question-answering. We validate XMedGPT across four pillars: multi-modal interpretability, uncertainty quantification, and prognostic modeling, and rigorous benchmarking. The model achieves an IoU of 0.703 across 141 anatomical regions, and a Kendall's tau-b of 0.479, demonstrating strong alignment between visual rationales and clinical outcomes. For uncertainty estimation, it attains an AUC of 0.862 on visual question answering and 0.764 on radiology report generation. In survival and recurrence prediction for lung and glioma cancers, it surpasses prior leading models by 26.9%, and outperforms GPT-4o by 25.0%. Rigorous benchmarking across 347 datasets covers 40 imaging modalities and external validation spans 4 anatomical systems confirming exceptional generalizability, with performance gains surpassing existing GMAI by 20.7% for in-domain evaluation and 16.7% on 11,530 in-house data evaluation. Together, XMedGPT represents a significant leap forward in clinician-centric AI integration, offering trustworthy and scalable support for diverse healthcare applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。