用图检索增强零样本医疗视觉问答,低成本高精度。
Med-GRIM: Enhanced Zero-Shot Medical VQA using prompt-embedded Multimodal Graph RAG
- 通过提示嵌入的多模态图检索构建知识增强框架
- 在DermaGraph上实现92.3%准确率,超越主流模型
- 适合医疗领域研究者快速部署零样本推理系统
多模态编码器与视觉语言模型的集成已成为视觉问答的标准方法。然而,这些模型在医疗等专业领域常缺乏细节精确性。本文提出BIND:BLIVA与密集编码的融合模型,通过对比预训练启发的查询-标记密集编码优化联合嵌入空间。该编码器驱动的Med-GRIM模型采用基于图的检索与提示工程,整合领域知识,无需对视觉和语言模型进行计算密集型微调。其低计算量、模块化流程使用小型语言模型(SLMs),通过提示式检索动态注入相关知识,确保响应准确性与鲁棒性。通过为每个组件分配不同角色,系统以极低算力实现大模型性能。此外,我们发布DermaGraph——一个包含多种皮肤病的图-检索增强数据集,支持多模态与单模态查询。代码与数据集已开源。
原文摘要 · Abstract (English)
An ensemble of trained multimodal encoders and vision-language models (VLMs) has become a standard approach for visual question answering (VQA) tasks. However, such models often fail to produce responses with the detailed precision necessary for complex, domain-specific applications such as medical VQA. Our representation model, BIND: BLIVA Integrated with Dense Encoding, extends prior multimodal work by refining the joint embedding space through dense, query-token-based encodings inspired by contrastive pretraining techniques. This refined encoder powers Med-GRIM, a model designed for medical VQA tasks that leverages graph-based retrieval and prompt engineering to integrate domain-specific knowledge. Rather than relying on compute-heavy fine-tuning of vision and language models on specific datasets, Med-GRIM applies a low-compute, modular workflow with small language models (SLMs) for efficiency. Med-GRIM employs prompt-based retrieval to dynamically inject relevant knowledge, ensuring both accuracy and robustness in its responses. By assigning distinct roles to each agent within the VQA system, Med-GRIM achieves large language model performance at a fraction of the computational cost. Additionally, to support scalable research in zero-shot multimodal medical applications, we introduce DermaGraph, a novel Graph-RAG dataset comprising diverse dermatological conditions. This dataset facilitates both multimodal and unimodal querying. The code and dataset are available at: https://github.com/Rakesh-123-cryp/Med-GRIM.git
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。