通过四阶段智能体框架,提升医学影像分析的准确性与可靠性。
Route, Retrieve, Reflect, Repair: Self-Improving Agentic Framework for Visual Detection and Linguistic Reasoning in Medical Imaging
- 分阶段构建路由、检索、反思与修复四类智能体协同工作。
- 报告评分提升1.7-2.5分,检测准确率提高2.5-3.5个百分点。
- 无需微调即可增强模型鲁棒性,适合临床医疗场景使用。
医学影像分析日益依赖大型视觉语言模型(VLMs),但多数系统仍是单次处理的黑箱,对推理过程、安全性和空间定位控制有限。我们提出R^4框架,将医学影像工作流分解为四个协作智能体:路由器根据图像、患者病史和元数据生成任务与专业化提示;检索器结合示例记忆与pass@k采样,联合生成自由文本报告和边界框;反思器针对关键临床错误模式(否定、侧别、无支持声明、矛盾、遗漏发现、定位错误)批判每个报告-框对;修复器在目标约束下迭代修正叙述与空间输出,同时为未来案例筛选高质量示例。在胸部X光分析中,基于多个现代VLM主干网络,在报告生成与弱监督检测任务上,R^4相比强基线单VLM模型,平均提升LLM-as-a-Judge分数1.7-2.5点,mAP50提升2.5-3.5绝对百分点,且无需梯度微调。结果表明,智能体式路由、反思与修复可使强大但脆弱的VLM转化为更可靠、更具备空间接地性的临床影像解读工具。代码见:https://github.com/faiyazabdullah/MultimodalMedAgent
原文摘要 · Abstract (English)
Medical image analysis increasingly relies on large vision-language models (VLMs), yet most systems remain single-pass black boxes that offer limited control over reasoning, safety, and spatial grounding. We propose R^4, an agentic framework that decomposes medical imaging workflows into four coordinated agents: a Router that configures task- and specialization-aware prompts from the image, patient history, and metadata; a Retriever that uses exemplar memory and pass@k sampling to jointly generate free-text reports and bounding boxes; a Reflector that critiques each draft-box pair for key clinical error modes (negation, laterality, unsupported claims, contradictions, missing findings, and localization errors); and a Repairer that iteratively revises both narrative and spatial outputs under targeted constraints while curating high-quality exemplars for future cases. Instantiated on chest X-ray analysis with multiple modern VLM backbones and evaluated on report generation and weakly supervised detection, R^4 consistently boosts LLM-as-a-Judge scores by roughly +1.7-+2.5 points and mAP50 by +2.5-+3.5 absolute points over strong single-VLM baselines, without any gradient-based fine-tuning. These results show that agentic routing, reflection, and repair can turn strong but brittle VLMs into more reliable and better grounded tools for clinical image interpretation. Our code can be found at: https://github.com/faiyazabdullah/MultimodalMedAgent
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。