用高效微调的Mamba模型生成胸部X光报告,提升诊断效率。
EMRRG: Efficient Fine-Tuning Pre-trained X-ray Mamba Networks for Radiology Report Generation
- 基于状态空间模型的Mamba网络处理图像分块特征
- 部分LoRA微调使报告生成准确率显著提升
- 适合医学影像与自然语言生成交叉研究者
基于X光图像的医学报告生成(MRG)是人工智能在医疗领域的重要方向,可有效减轻临床医生负担并缩短患者等待时间。现有模型多依赖大语言模型(LLM),对预训练视觉基础模型或先进微调技术探索不足。主流框架或回避微调,或仅采用简单的LoRA方法,忽视了跨注意力机制的优化潜力。尽管Transformer主导视觉-语言任务,非Transformer架构如Mamba网络在医学报告生成中仍鲜有研究,具备广阔前景。本文提出EMRRG框架,通过参数高效方法微调预训练的Mamba网络。具体地,将X光图像划分为图像块,经基于SSM的视觉主干提取特征,采用部分LoRA实现最优性能;结合混合解码器的LLM生成报告,支持端到端训练,在三个主流基准数据集上验证了方法的有效性。代码已开源。
原文摘要 · Abstract (English)
X-ray image-based medical report generation (MRG) is a pivotal area in artificial intelligence that can significantly reduce diagnostic burdens for clinicians and patient wait times. Existing MRG models predominantly rely on Large Language Models (LLMs) to improve report generation, with limited exploration of pre-trained vision foundation models or advanced fine-tuning techniques. Mainstream frameworks either avoid fine-tuning or utilize simplistic methods like LoRA, often neglecting the potential of enhancing cross-attention mechanisms. Additionally, while Transformer-based models dominate vision-language tasks, non-Transformer architectures, such as the Mamba network, remain underexplored for medical report generation, presenting a promising avenue for future research. In this paper, we propose EMRRG, a novel X-ray report generation framework that fine-tunes pre-trained Mamba networks using parameter-efficient methods. Specifically, X-ray images are divided into patches, tokenized, and processed by an SSM-based vision backbone for feature extraction, with Partial LoRA yielding optimal performance. An LLM with a hybrid decoder generates the medical report, enabling end-to-end training and achieving strong results on benchmark datasets. Extensive experiments on three widely used benchmark datasets fully validated the effectiveness of our proposed strategies for the X-ray MRG. The source code of this paper will be released on https://github.com/Event-AHU/Medical_Image_Analysis.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。