用多个角色协作的AI系统，提升多模态医疗诊断准确率
MAM: Modular Multi-Agent Framework for Multi-Modal Medical Diagnosis via Role-Specialized Collaboration
- 将医生角色拆分为全科、专科、影像等专家，分头处理不同任务
- 在多模态数据上测试，性能比基线模型高出18%至365%
- 适合需要灵活更新知识的医疗AI研发团队使用
近期医学大语言模型在推理与诊断能力方面展现出强大潜力。然而，现有统一的多模态医学LLM在知识更新成本、覆盖全面性和灵活性方面存在局限。为此，我们提出模块化多智能体医疗诊断框架MAM。基于实证发现的角色分工与诊断区分优势，MAM将诊断过程分解为五个角色：全科医生、专科团队、放射科医生、医疗助理和主任，每个角色由一个基于LLM的智能体承担。该模块化协同框架支持高效的知识更新，并可利用已有医学LLM与知识库。在包含文本、图像、音频和视频等多种模态的公开多模态医疗数据集上进行的广泛实验表明，MAM在各项指标上均显著优于单一模态的LLM。值得注意的是，相较于基线模型，其性能提升幅度达18%至365%。代码已开源：https://github.com/yczhou001/MAM。
原文摘要 · Abstract (English)
Recent advancements in medical Large Language Models (LLMs) have showcased their powerful reasoning and diagnostic capabilities. Despite their success, current unified multimodal medical LLMs face limitations in knowledge update costs, comprehensiveness, and flexibility. To address these challenges, we introduce the Modular Multi-Agent Framework for Multi-Modal Medical Diagnosis (MAM). Inspired by our empirical findings highlighting the benefits of role assignment and diagnostic discernment in LLMs, MAM decomposes the medical diagnostic process into specialized roles: a General Practitioner, Specialist Team, Radiologist, Medical Assistant, and Director, each embodied by an LLM-based agent. This modular and collaborative framework enables efficient knowledge updates and leverages existing medical LLMs and knowledge bases. Extensive experimental evaluations conducted on a wide range of publicly accessible multimodal medical datasets, incorporating text, image, audio, and video modalities, demonstrate that MAM consistently surpasses the performance of modality-specific LLMs. Notably, MAM achieves significant performance improvements ranging from 18% to 365% compared to baseline models. Our code is released at https://github.com/yczhou001/MAM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。