用3D视觉模型与大语言模型结合，自动生成医学影像报告
ViT3D Alignment of LLaMA3: 3D Medical Image Report Generation
- 用ViT3D处理3D医学影像，用LLaMA3生成报告
- 报告生成得分0.3，视觉问答准确率0.61，优于基线
- 小数据微调即可有效，适合医疗文本生成场景
自动医学报告生成（MRG）旨在从医学影像中生成详细文本报告，已成为该领域的重要任务。此类系统可通过减少放射科医生的报告撰写时间与精力，提升诊断效率。本文提出一种基于多模态大语言模型的MRG新方法。具体而言，采用来自M3D-CLIP的3D视觉变换器（ViT3D）作为图像编码器处理3D扫描，使用Asclepius-Llama3-8B作为语言模型通过自回归解码生成文本报告。实验显示，本模型在MRG任务验证集上平均Green得分为0.3，在视觉问答（VQA）任务验证集上平均准确率为0.61，优于基线模型。结果表明，通过在小数据集上微调，ViT3D与LLaMA3的对齐在自动MRG和VQA任务中均具有效性。
原文摘要 · Abstract (English)
Automatic medical report generation (MRG), which aims to produce detailed text reports from medical images, has emerged as a critical task in this domain. MRG systems can enhance radiological workflows by reducing the time and effort required for report writing, thereby improving diagnostic efficiency. In this work, we present a novel approach for automatic MRG utilizing a multimodal large language model. Specifically, we employed the 3D Vision Transformer (ViT3D) image encoder introduced from M3D-CLIP to process 3D scans and use the Asclepius-Llama3-8B as the language model to generate the text reports by auto-regressive decoding. The experiment shows our model achieved an average Green score of 0.3 on the MRG task validation set and an average accuracy of 0.61 on the visual question answering (VQA) task validation set, outperforming the baseline model. Our approach demonstrates the effectiveness of the ViT3D alignment of LLaMA3 for automatic MRG and VQA tasks by tuning the model on a small dataset.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。