arXiv:2608.21810cs.LGcs.CV2026-08

让医疗AI像医生一样积累经验,持续提升诊断能力。

MSM-Mem: A Universal Medical Structured Multimodal Memory Framework for Medical AI Agents

论文配图:MSM-Mem: A Universal Medical Structured Multimodal Memory Framework for Medical AI Agents
图 1 · 摘自论文原文
  • 将临床经验分为语义、事件和视觉三类记忆,动态更新
  • 在多模态大模型上使用后,性能随使用次数持续提升
  • 适合需要长期学习与个性化推理的智能诊疗系统

临床决策本质上依赖经验:医生通过整合患者病史、多模态观察及过往诊断经验,在多次交互中逐步优化判断。然而,当前基于多模态大语言模型(MLLM)的医疗AI代理大多为无状态推理系统,每次交互独立生成决策,缺乏对经验知识的保留与内化。这限制了其通过使用过程提升推理可靠性,并难以适应真实临床场景中的纵向患者背景。本文提出医学结构化多模态记忆框架MSM-Mem,使医疗AI代理能通过累积临床经验不断进化。该框架将异构临床经验组织为语义、情景和视觉记忆,并在推理过程中增量更新,支持代理检索过往经验以指导当前推理,实现决策能力的渐进式优化。在MoE-LLaVA基线上的评估显示,随着持续使用,性能持续提升。MSM-Mem为构建可像医生一样通过实践不断学习的医疗AI代理提供了可行路径。

原文摘要 · Abstract (English)

Clinical decision-making is inherently experience-driven: physicians progressively refine their reasoning by synthesizing patient history, multimodal observations, and prior diagnostic experiences across interactions. In contrast, current multimodal large language model (MLLM)-based medical AI agents largely operate as stateless inference systems, generating decisions independently for each interaction without retaining or internalizing experiential knowledge. This discrepancy limits their ability to progressively improve reasoning reliability through usage and adapt to longitudinal patient contexts in real-world clinical workflows. In this study, we propose Medical Structured Multimodal Memory (MSM-Mem), an agentic memory framework that enables medical AI agents to evolve through accumulated clinical experiences. MSM-Mem organizes heterogeneous clinical experiences into semantic, episodic, and visual memory and incrementally updates them during inference, allowing the agent to retrieve prior experiences to inform current reasoning and progressively refine decision-making over time. Evaluations on MoE-LLaVA backbones demonstrate consistent performance improve- ments with further gains observed through continued usage. In general, MSM-Mem offers a viable pathway toward medical AI agents capable of evolving their reasoning competence in a manner analogous to the way clinicians learn from practice over time.

医疗AI多模态记忆推理优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。