用生成的报告增强胸部X光问答模型,提升对异常和变化的识别能力。
Grounding Chest X-Ray Visual Question Answering with Generated Radiology Reports
- 分两步生成:先预测报告,再基于报告回答问题。
- 在医学差异问答数据集上达到当前最好效果。
- 适合医疗AI研究者和临床辅助系统开发者。
我们提出一种新型胸部X光图像问答方法,同时处理单图异常识别和纵向对比差异问题。对于单图问题(如‘图像X中有哪些异常?’),模型输入一张胸片;对于差异问题(如‘图像X与Y有何不同?’),输入同一患者不同时期的两张胸片,以检测时间变化。受思维链启发,我们发现将为同一胸片生成的放射科报告作为额外证据输入答案生成模块,可显著提升性能。模型分为两阶段:一是报告生成(RG),二是答案生成(AG)。实验表明,融合预测报告能有效提升单图与差异问答任务表现,在Medical-Diff-VQA数据集上取得当前最优结果。
原文摘要 · Abstract (English)
We present a novel approach to Chest X-ray (CXR) Visual Question Answering (VQA), addressing both single-image image-difference questions. Single-image questions focus on abnormalities within a specific CXR ("What abnormalities are seen in image X?"), while image-difference questions compare two longitudinal CXRs acquired at different time points ("What are the differences between image X and Y?"). We further explore how the integration of radiology reports can enhance the performance of VQA models. While previous approaches have demonstrated the utility of radiology reports during the pre-training phase, we extend this idea by showing that the reports can also be leveraged as additional input to improve the VQA model's predicted answers. First, we propose a unified method that handles both types of questions and auto-regressively generates the answers. For single-image questions, the model is provided with a single CXR. For image-difference questions, the model is provided with two CXRs from the same patient, captured at different time points, enabling the model to detect and describe temporal changes. Taking inspiration from 'Chain-of-Thought reasoning', we demonstrate that performance on the CXR VQA task can be improved by grounding the answer generator module with a radiology report predicted for the same CXR. In our approach, the VQA model is divided into two steps: i) Report Generation (RG) and ii) Answer Generation (AG). Our results demonstrate that incorporating predicted radiology reports as evidence to the AG model enhances performance on both single-image and image-difference questions, achieving state-of-the-art results on the Medical-Diff-VQA dataset.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。