用智能体框架解决大模型眼底图像误诊与不稳定问题
An Agentic AI Framework Overcomes Fundamental Limitations of Large Language Models for Glaucoma Detection from Fundus Photography
- 将大模型与专业工具联动，分三步完成诊断评估
- 准确率提升16至47个百分点，接近专家水平
- 适合医疗AI开发者及希望提升诊断稳定性的研究者
大型语言模型（LLMs）在医学影像解读中展现潜力，但存在幻觉、准确率低和运行间不一致等问题。我们开发并验证了一种智能体AI框架，将LLM与专用深度学习工具结合，用于从眼底照片检测青光眼。流程分为三步：(1) LLM初步评估；(2) 调用专用工具进行图像质量评估（QAModel, FundaQ-8）、青光眼分类（SwinV2-Tiny）和视盘/杯分割（SegFormer-B0）；(3) LLM反思整合初始判断与工具输出。在两个公开数据集（ORIGA, n=100；RIM-ONE-v3, n=100）上，对未裁剪与裁剪视野分别测试，所有图像由盲法训练的青光眼专科医生独立评分。智能体流程在所有条件下使分类准确率提升16至47个百分点，最接近专家水平（相差仅6点）；在RIM-ONE-v3上最优配置达到专家准确率88%。单独使用LLM时表现失败：GPT-5.4 mini出现明显假阳性（敏感性95-100%，特异性0-5%），Gemini 2.5 Flash则运行间随机波动；智能体框架有效纠正两类问题。杯盘比误差降低15%-50%（平均绝对误差从0.156-0.228降至0.104-0.132），与专家评分相关性由弱（r=0.12-0.39）升至中到强（r=0.59-0.84）。运行一致性从近乎随机（κ最低-0.01）提升至近乎完美（κ最高0.96）。集成专用工具可有效克服纯LLM方法的关键缺陷，包括过度诊断和运行不一致。性能提升在两种LLM上均成立，表明跨模型通用性，可能预示医疗AI从单一模型向协同多智能体系统转变。
原文摘要 · Abstract (English)
Large language models (LLMs) show promise in medical image interpretation but suffer from hallucination, limited accuracy, and run-to-run inconsistency. We developed and validated an agentic AI framework integrating LLMs with specialized deep learning tools for glaucoma detection from fundus photography. The workflow had three steps: (1) LLM initial assessment; (2) function calling to invoke specialized tools for image quality (QAModel, FundaQ-8), glaucoma classification (SwinV2-Tiny), and optic disc/cup segmentation (SegFormer-B0); and (3) LLM reflection integrating the initial impression with tool outputs. Two LLMs (Gemini 2.5 Flash, GPT-5.4 mini) were evaluated on two public datasets (ORIGA, n=100; RIM-ONE-v3, n=100) under uncropped and cropped fields of view; all images were independently graded by a masked fellowship-trained glaucoma specialist. The agentic workflow improved classification accuracy by 16 to 47 percentage points across all conditions, reaching within 6 points of the specialist; on RIM-ONE-v3 the best configurations matched the specialist accuracy of 88%. LLM-alone approaches failed in two ways: GPT-5.4 mini showed positive bias (sensitivity 95-100%, specificity 0-5%), while Gemini 2.5 Flash varied stochastically between runs; the agentic workflow corrected both. Cup-to-disc ratio error fell 15-50% (MAE 0.156-0.228 to 0.104-0.132), and correlation with specialist grading rose from weak (r=0.12-0.39) to moderate-strong (r=0.59-0.84). Run-to-run consistency rose from near-random (kappa as low as -0.01) to near-perfect (kappa up to 0.96). Integrating LLMs with specialized tools addressed key limitations of LLM-alone approaches, including over-diagnosis and run-to-run variability. Gains held for both LLMs, suggesting generalizability across backbones, and may signal a shift from monolithic models toward orchestrated multi-agent systems in medical AI.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。