DermAgent通过多工具协作实现可追溯的皮肤图像诊断,减少幻觉并提升准确率。
DermAgent: A Self-Reflective Agentic System for Dermatological Image Analysis with Multi-Tool Reasoning and Traceable Decision-Making

- 构建Plan-Execute-Reflect框架,融合七类视觉语言模块进行分步推理。
- 在五项基准上超越GPT-4o,疾病诊断准确率提升17.6%,摘要生成ROUGE-L提高3.15%。
- 适合需要高可信度医学影像分析的研究者与临床辅助系统开发者。
皮肤科诊断需结合精细视觉感知与专家临床知识。尽管多模态大语言模型(MLLMs)促进交互式医学图像分析,但在皮肤科应用中受限于领域知识不足和幻觉问题。为此,我们提出DermAgent,一个协同多工具代理,基于计划-执行-反思框架,协调七个专用视觉与语言模块。该系统通过三大核心组件实现逐步、可追溯的诊断推理:首先,使用互补视觉感知工具完成形态描述、皮肤镜概念标注与疾病诊断;其次,为弥补领域先验缺失,设计双模态检索模块,通过交叉比对413,210个已诊断图像案例与3,199条临床指南片段,将每项预测锚定于外部证据;再次,引入确定性批评模块,通过置信度、覆盖率与冲突门控进行严格事后审计,自动检测跨源不一致并触发针对性自我修正。在五个皮肤科基准上的实验表明,DermAgent在零样本细粒度疾病诊断、概念标注与临床描述任务中持续优于最先进MLLMs与医疗代理基线,疾病诊断准确率较GPT-4o提升17.6%,摘要生成的ROUGE-L指标提高3.15%。代码已开源于https://github.com/YizeezLiu/DermAgent。
原文摘要 · Abstract (English)
Dermatological diagnosis requires integrating fine-grained visual perception with expert clinical knowledge. Although Multimodal Large Language Models (MLLMs) facilitate interactive medical image analysis, their application in dermatology is hindered by insufficient domain-specific grounding and hallucinations. To address these issues, we propose DermAgent, a collaborative multi-tool agent that orchestrates seven specialized vision and language modules within a Plan-Execute-Reflect framework. DermAgent delivers stepwise, traceable diagnostic reasoning through three core components. First, it employs complementary visual perception tools for comprehensive morphological description, dermoscopic concept annotation, and disease diagnosis. Second, to overcome the lack of domain prior, a dual-modality retrieval module anchors every prediction in external evidence by cross-referencing 413,210 diagnosed image cases and 3,199 clinical guideline chunks. To further mitigate hallucinations, a deterministic critic module conducts strict post-hoc auditing via confidence, coverage, and conflict gates, automatically detecting inter-source disagreements to trigger targeted self-correction. Extensive experiments on five dermatology benchmarks demonstrate that DermAgent consistently outperforms state-of-the-art MLLMs and medical agent baselines across zero-shot fine-grained disease diagnosis, concept annotation, and clinical captioning tasks, exceeding GPT-4o by 17.6% in skin disease diagnostic accuracy and 3.15% in captioning ROUGE-L. Our code is available at https://github.com/YizeezLiu/DermAgent.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。