arXiv:2411.16523cs.CVcs.CL2024-11被引 6

用标签增强检索生成,小模型也能高效生成放射科报告。

LaB-RAG: Label Boosted Retrieval Augmented Generation for Radiology Report Generation

  • 用分类模型将X光片转为放射科标签,驱动大语言模型生成
  • 在MIMIC-CXR和CheXpert Plus上超越其他检索方法
  • 无需微调语言或图像模型,适合医疗场景快速部署

当前图像描述任务依赖深度学习模型从图像嵌入中生成文本。本文挑战了需微调大型专用模型以提升生成精度的假设,提出标签增强检索增强生成(LaB-RAG)方法,基于小模型利用类别标签增强预训练大语言模型的检索增强生成能力。在MIMIC-CXR和CheXpert Plus数据集上的放射科报告生成任务中,我们证明:简单分类模型结合零样本嵌入可将胸部X光片有效转化为放射科特定标签,并与标准RAG结合,使通用领域大语言模型生成高质量报告。无需对生成模型或图像嵌入模型进行任务特化训练,也无需直接向大语言模型输入影像,LaB-RAG在自然语言与放射科语言指标上均优于其他检索基线方法,且与微调型视觉-语言模型表现相当。通过大量消融实验,验证了各组件有效性及与微调方法的兼容性,可进一步提升报告生成性能。

原文摘要 · Abstract (English)

In the current paradigm of image captioning, deep learning models are trained to generate text from image embeddings of latent features. We challenge the assumption that fine-tuning of large, bespoke models is required to improve model generation accuracy. Here we propose Label Boosted Retrieval Augmented Generation (LaB-RAG), a small-model-based approach to image captioning that leverages image descriptors in the form of categorical labels to boost standard retrieval augmented generation (RAG) with pretrained large language models (LLMs). We study our method in the context of radiology report generation (RRG) over MIMIC-CXR and CheXpert Plus. We argue that simple classification models combined with zero-shot embeddings can effectively transform X-rays into text-space as radiology-specific labels. In combination with standard RAG, we show that these derived text labels can be used with general-domain LLMs to generate radiology reports. Without ever training our generative language model or image embedding models specifically for the task, and without ever directly "showing" the LLM an X-ray, we demonstrate that LaB-RAG achieves better results across natural language and radiology language metrics compared with other retrieval-based RRG methods, while attaining competitive results compared to other fine-tuned vision-language RRG models. We further conduct extensive ablation experiments to better understand the components of LaB-RAG. Our results suggest broader compatibility and synergy with fine-tuned methods to further enhance RRG performance.

放射科报告检索增强小模型标签增强

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。