用相似度引导注意力,让医学影像理解更可解释且支持零样本多任务
RadZero: Similarity-Based Cross-Attention for Explainable Vision-Language Alignment in Chest X-ray with Zero-Shot Multi-Task Capability
- 基于文本与图像局部特征的相似度计算,实现细粒度视觉语言对齐
- 在零样本分类、定位和分割任务上超越现有方法,定位精度提升5.2%以上
- 适合医疗AI可解释性研究者,也适用于需要开放词汇语义分割的临床场景
近年来多模态模型显著提升了放射科影像中的视觉-语言(VL)对齐能力。然而,现有方法难以有效利用复杂的放射科报告进行学习,且通过注意力概率可视化提供的可解释性有限。为此,我们提出新型框架RadZero,具备零样本多任务能力,核心为基于相似度的视觉-语言跨注意力机制(VL-CABS)。该机制将文本嵌入与局部图像特征对齐,实现可解释的细粒度推理。RadZero利用大语言模型从报告中提取简洁语义句,并采用多正例对比学习捕捉图像与多个相关描述之间的关系。其采用预训练视觉编码器外接可训练Transformer层,支持高效高分辨率图像处理。通过计算文本嵌入与局部图像块特征的相似度,VL-CABS实现了零样本分类的相似度概率,以及像素级的视觉-语言相似度图,用于定位与分割。在公开胸部X光数据集上的实验表明,RadZero在零样本分类、定位和分割任务上均优于当前最优方法。此外,相似度图分析揭示了其在提升可解释性方面的潜力。定性评估还验证了其在开放词汇语义分割中的有效性。代码已开源。
原文摘要 · Abstract (English)
Recent advancements in multimodal models have significantly improved vision-language (VL) alignment in radiology. However, existing approaches struggle to effectively utilize complex radiology reports for learning and offer limited interpretability through attention probability visualizations. To address these challenges, we introduce $\textbf{RadZero}$, a novel framework for VL alignment in chest X-ray with zero-shot multi-task capability. A key component of our approach is $\textbf{VL-CABS}$ ($\textbf{V}$ision-$\textbf{L}$anguage $\textbf{C}$ross-$\textbf{A}$ttention $\textbf{B}$ased on $\textbf{S}$imilarity), which aligns text embeddings with local image features for interpretable, fine-grained VL reasoning. RadZero leverages large language models to extract concise semantic sentences from radiology reports and employs multi-positive contrastive training to effectively capture relationships between images and multiple relevant textual descriptions. It uses a pre-trained vision encoder with additional trainable Transformer layers, allowing efficient high-resolution image processing. By computing similarity between text embeddings and local image patch features, VL-CABS enables zero-shot inference with similarity probability for classification, and pixel-level VL similarity maps for grounding and segmentation. Experimental results on public chest radiograph benchmarks show that RadZero outperforms state-of-the-art methods in zero-shot classification, grounding, and segmentation. Furthermore, VL similarity map analysis highlights the potential of VL-CABS for improving explainability in VL alignment. Additionally, qualitative evaluation demonstrates RadZero's capability for open-vocabulary semantic segmentation, further validating its effectiveness in medical imaging. Code is available at $\href{https://github.com/deepnoid-ai/RadZero}{https://github.com/deepnoid-ai/RadZero}$.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。