通过异常感知反馈提升医学视觉语言模型对病灶的识别能力
Improving Medical Large Vision-Language Models with Abnormal-Aware Feedback
- 构建异常揭示数据集，用GPT-4V生成诊断辅助标注
- 两阶段训练使异常定位准确率提升58%以上
- 适合医学AI研发与临床辅助诊断系统开发者
现有医学大视觉语言模型（Med-LVLMs）虽具备强大医学图像理解能力，但在病灶定位方面仍存在挑战。为此，本文提出UMed-LVLM模型，旨在增强对医学异常的识别能力。研究构建了医学异常揭示（MAU）数据集，利用GPT-4V根据图像中异常区域生成诊断以辅助标注。提出两阶段训练方法：异常感知指令微调与异常感知奖励机制，包含相关性奖励、异常定位奖励和视觉相关性奖励。实验表明，本方法在异常识别与理解上显著优于现有模型，性能相比基线提升58%。同时验证了强化异常检测能力可有效提升模型对医学图像的理解力与泛化性能。
原文摘要 · Abstract (English)
Existing Medical Large Vision-Language Models (Med-LVLMs), encapsulating extensive medical knowledge, demonstrate excellent capabilities in understanding medical images. However, there remain challenges in visual localization in medical images, which is crucial for abnormality detection and interpretation. To address these issues, we propose a novel UMed-LVLM designed to unveil medical abnormalities. Specifically, we collect a Medical Abnormalities Unveiling (MAU) dataset and propose a two-stage training method for UMed-LVLM training. To collect MAU dataset, we propose a prompt method utilizing the GPT-4V to generate diagnoses based on identified abnormal areas in medical images. Moreover, the two-stage training method includes Abnormal-Aware Instruction Tuning and Abnormal-Aware Rewarding, comprising Relevance Reward, Abnormal Localization Reward and Vision Relevance Reward. Experimental results demonstrate that our UMed-LVLM significantly outperforms existing Med-LVLMs in identifying and understanding medical abnormalities, achieving a 58% improvement over the baseline. In addition, this work shows that enhancing the abnormality detection capabilities of Med-LVLMs significantly improves their understanding of medical images and generalization capability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。