将深度学习与大模型结合,让医学影像系统能生成可靠的临床推理报告。
DL$^3$M: A Vision-to-Language Framework for Expert-Level Medical Reasoning through Deep Learning and Large Language Models
- 用MobileCoAtNet模型精准识别胃部八类疾病
- 结合专家标注数据集,验证大模型推理稳定性不足
- 为医疗AI可解释性提供可复现的评估框架
医学图像分类器能有效检测消化道疾病,但无法解释决策依据。大语言模型虽可生成临床文本,却在视觉推理中表现不稳定且常出错。为此,我们提出一个连接图像分类与结构化临床推理的框架。设计了针对内镜图像的新型混合模型MobileCoAtNet,对八类胃部疾病实现高精度分类。其输出驱动多个大语言模型进行推理,并构建两个经专家验证的基准数据集,涵盖病因、症状、治疗、生活方式及随访等维度。对32个大语言模型进行评估发现,强分类能力提升解释质量,但无一达到人类水平的稳定性——即使最优模型在不同提示下也会改变推理结论。研究显示,深度学习与大模型结合可生成有用临床叙事,但当前大模型仍不可靠用于高风险医疗决策。该框架揭示了其局限性,也为构建更安全的推理系统指明方向。完整代码与数据集见https://github.com/souravbasakshuvo/DL3M。
原文摘要 · Abstract (English)
Medical image classifiers detect gastrointestinal diseases well, but they do not explain their decisions. Large language models can generate clinical text, yet they struggle with visual reasoning and often produce unstable or incorrect explanations. This leaves a gap between what a model sees and the type of reasoning a clinician expects. We introduce a framework that links image classification with structured clinical reasoning. A new hybrid model, MobileCoAtNet, is designed for endoscopic images and achieves high accuracy across eight stomach-related classes. Its outputs are then used to drive reasoning by several LLMs. To judge this reasoning, we build two expert-verified benchmarks covering causes, symptoms, treatment, lifestyle, and follow-up care. Thirty-two LLMs are evaluated against these gold standards. Strong classification improves the quality of their explanations, but none of the models reach human-level stability. Even the best LLMs change their reasoning when prompts vary. Our study shows that combining DL with LLMs can produce useful clinical narratives, but current LLMs remain unreliable for high-stakes medical decisions. The framework provides a clearer view of their limits and a path for building safer reasoning systems. The complete source code and datasets used in this study are available at https://github.com/souravbasakshuvo/DL3M.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。