用大模型模拟糖尿病视网膜病变筛查中的AI辅助，测试不同输出形式的效果。
Simulating Clinical AI Assistance using Multimodal LLMs: A Case Study in Diabetic Retinopathy
- 用多模态大模型模拟医生与AI协作，测试不同输出格式的影响。
- 医学专用模型MedGemma在敏感性和准确率上优于通用模型GPT-4o。
- 描述性输出能提升可解释性，适合临床信任和资源匮乏环境。
糖尿病视网膜病变（DR）是全球致盲主因之一，AI系统可拓展眼底摄影筛查的可及性。当前获FDA认证的系统主要提供二分类转诊结果，这种极简输出可能限制临床信任与实用性。然而，何种输出形式最有效提升医工协作，尚缺乏大规模实证研究。本文评估了多模态大语言模型（MLLMs）在DR检测中的表现及其模拟临床AI辅助的能力。在IDRiD和Messidor-2数据集上测试了GPT-4o（通用模型）和MedGemma（开源医学模型）。实验包括：(1) 基线评估，(2) 基于合成预测的模拟辅助，(3) 实际的AI-to-AI协作（GPT-4o融合MedGemma输出）。结果显示，MedGemma基线性能更优，敏感性与AUROC更高；而GPT-4o特异性接近完美但敏感性低。两者均可根据模拟输入调整判断，但GPT-4o对错误输入反应剧烈，而MedGemma更稳定。在真实协作中，即使未直接访问图像，仅依赖MedGemma的描述性输出，GPT-4o仍实现极高性能（最高AUROC达0.96）。结果表明，MLLMs可优化DR筛查流程，并作为高效模拟器研究不同输出配置下的临床辅助效果。开放、轻量的MedGemma模型尤其适用于低资源环境，描述性输出则有助于提升可解释性与临床信任。
原文摘要 · Abstract (English)
Diabetic retinopathy (DR) is a leading cause of blindness worldwide, and AI systems can expand access to fundus photography screening. Current FDA-cleared systems primarily provide binary referral outputs, where this minimal output may limit clinical trust and utility. Yet, determining the most effective output format to enhance clinician-AI performance is an empirical challenge that is difficult to assess at scale. We evaluated multimodal large language models (MLLMs) for DR detection and their ability to simulate clinical AI assistance across different output types. Two models were tested on IDRiD and Messidor-2: GPT-4o, a general-purpose MLLM, and MedGemma, an open-source medical model. Experiments included: (1) baseline evaluation, (2) simulated AI assistance with synthetic predictions, and (3) actual AI-to-AI collaboration where GPT-4o incorporated MedGemma outputs. MedGemma outperformed GPT-4o at baseline, achieving higher sensitivity and AUROC, while GPT-4o showed near-perfect specificity but low sensitivity. Both models adjusted predictions based on simulated AI inputs, but GPT-4o's performance collapsed with incorrect ones, whereas MedGemma remained more stable. In actual collaboration, GPT-4o achieved strong results when guided by MedGemma's descriptive outputs, even without direct image access (AUROC up to 0.96). These findings suggest MLLMs may improve DR screening pipelines and serve as scalable simulators for studying clinical AI assistance across varying output configurations. Open, lightweight models such as MedGemma may be especially valuable in low-resource settings, while descriptive outputs could enhance explainability and clinician trust in clinical workflows.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。