用强化学习提升医疗视觉语言模型的推理能力与泛化性,兼顾低资源部署。
RARL: Improving Medical VLM Reasoning and Generalization with Reinforcement Learning and LoRA under Data and Hardware Constraints
- 基于LoRA和自定义奖励函数,通过强化学习优化医疗VLM推理能力。
- 在单张A100 GPU上训练,推理任务准确率提升7.78%,泛化性能提高27%。
- 适合关注医疗AI可解释性与轻量化部署的研究者与开发者。
视觉语言模型(VLMs)在医疗诊断中的应用日益广泛,但现有模型普遍存在泛化能力差、透明度低和计算效率不足的问题,限制了其在资源受限环境下的实际部署。为此,我们提出一种面向推理的强化学习框架RARL,通过低秩适配(LoRA)对轻量级基础模型Qwen2-VL-2B-Instruct进行微调,并设计融合诊断准确率与推理质量的定制奖励函数。所有训练仅在单块NVIDIA A100-PCIE-40GB GPU上完成,验证了其在低资源环境下的可行性。采用大语言模型作为评判者评估正确性与解释质量。实验表明,相较于监督微调,RARL在推理任务上提升约7.78%;在未见数据集上,泛化性能较监督微调提升约27%,较传统强化学习微调提升约4%。此外,训练时使用多样性提示、推理时使用推理提示对性能提升至关重要。结果表明,基于推理引导的学习与提示策略能有效推动医疗VLM向更透明、准确、高效的方向发展。代码与数据已公开。
原文摘要 · Abstract (English)
The growing integration of vision-language models (VLMs) in medical applications offers promising support for diagnostic reasoning. However, current medical VLMs often face limitations in generalization, transparency, and computational efficiency-barriers that hinder deployment in real-world, resource-constrained settings. To address these challenges, we propose a Reasoning-Aware Reinforcement Learning framework, \textbf{RARL}, that enhances the reasoning capabilities of medical VLMs while remaining efficient and adaptable to low-resource environments. Our approach fine-tunes a lightweight base model, Qwen2-VL-2B-Instruct, using Low-Rank Adaptation and custom reward functions that jointly consider diagnostic accuracy and reasoning quality. Training is performed on a single NVIDIA A100-PCIE-40GB GPU, demonstrating the feasibility of deploying such models in constrained environments. We evaluate the model using an LLM-as-judge framework that scores both correctness and explanation quality. Experimental results show that RARL significantly improves VLM performance in medical image analysis and clinical reasoning, outperforming supervised fine-tuning on reasoning-focused tasks by approximately 7.78%, while requiring fewer computational resources. Additionally, we demonstrate the generalization capabilities of our approach on unseen datasets, achieving around 27% improved performance compared to supervised fine-tuning and about 4% over traditional RL fine-tuning. Our experiments also illustrate that diversity prompting during training and reasoning prompting during inference are crucial for enhancing VLM performance. Our findings highlight the potential of reasoning-guided learning and reasoning prompting to steer medical VLMs toward more transparent, accurate, and resource-efficient clinical decision-making. Code and data are publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。