构建医疗多模态对话评估基准，验证大模型真实问诊能力
3MDBench: Medical Multimodal Multi-agent Dialogue Benchmark
- 用性格化患者代理和评估代理模拟真实远程问诊场景
- 多模态对话加内部推理使诊断F1提升6.5%，引入诊断CNN预测可再增20%
- 适合研究医疗AI对话、多模态诊断的开发者与临床研究者
尽管大型视觉语言模型（LVLMs）在医学领域受到广泛关注，但其在结合准确诊断与专业对话的复杂远程诊疗中的表现仍缺乏系统评估。本文提出3MDBench（医学多模态多智能体对话基准），一个开源框架，用于模拟和评估基于LVLM的远程诊疗过程。该基准通过基于性格的患者代理模拟患者多样性，并通过评估代理衡量诊断准确性和对话质量。数据集包含来自真实远程医疗交互的2996个案例，覆盖34种诊断，融合文本与图像数据。实验对比了主流开源与闭源LVLM的诊断策略。结果表明，结合内部推理的多模态对话相比非对话设置，F1分数提升6.5%，凸显上下文感知、信息获取式提问的重要性；此外，在LVLM上下文中注入诊断卷积神经网络的预测结果，最高可使F1提升20%。代码已开源：https://github.com/univanxx/3mdbench。
原文摘要 · Abstract (English)
Though Large Vision-Language Models (LVLMs) are being actively explored in medicine, their ability to conduct complex real-world telemedicine consultations combining accurate diagnosis with professional dialogue remains underexplored. This paper presents 3MDBench (Medical Multimodal Multi-agent Dialogue Benchmark), an open-source framework for simulating and evaluating LVLM-driven telemedical consultations. 3MDBench simulates patient variability through temperament-based Patient Agent and evaluates diagnostic accuracy and dialogue quality via Assessor Agent. It includes 2996 cases across 34 diagnoses from real-world telemedicine interactions, combining textual and image-based data. The experimental study compares diagnostic strategies for widely used open and closed-source LVLMs. We demonstrate that multimodal dialogue with internal reasoning improves F1 score by 6.5% over non-dialogue settings, highlighting the importance of context-aware, information-seeking questioning. Moreover, injecting predictions from a diagnostic convolutional neural network into the LVLM's context boosts F1 by up to 20%. Source code is available at https://github.com/univanxx/3mdbench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。