提升医学图文问答的精准度，通过对齐、挖掘难点样本和精选知识实现
Alignment, Mining and Fusion: Representation Alignment with Hard Negative Mining and Selective Knowledge Fusion for Medical Visual Question Answering
- 多层级跨模态对齐结合对比学习与最优传输理论
- 利用软标签挖掘难负样本并强化区分能力
- 门控交叉注意力融合答案词典，只选相关知识
医学视觉问答（Med-VQA）需同时理解医学图像与文本问题。尽管基于医学多模态预训练的方法在该任务上表现优异，但跨模态对齐尚无统一方案，难负样本问题仍被忽视，且常用知识融合方法可能引入无关信息。本文提出新框架：(1) 采用对比学习与最优传输理论，在多层级、多模态、多视角、多阶段实现统一的异构模态对齐；(2) 提出一种基于软标签的难负样本挖掘方法，增强多模态对齐中的负样本判别力；(3) 设计门控交叉注意力模块，将答案词汇表作为先验知识，选择性融合相关上下文。在RAD-VQA、SLAKE、PathVQA和VQA-2019等主流数据集上，本方法优于现有最先进模型。
原文摘要 · Abstract (English)
Medical Visual Question Answering (Med-VQA) is a challenging task that requires a deep understanding of both medical images and textual questions. Although recent works leveraging Medical Vision-Language Pre-training (Med-VLP) have shown strong performance on the Med-VQA task, there is still no unified solution for modality alignment, and the issue of hard negatives remains under-explored. Additionally, commonly used knowledge fusion techniques for Med-VQA may introduce irrelevant information. In this work, we propose a framework to address these challenges through three key contributions: (1) a unified solution for heterogeneous modality alignments across multiple levels, modalities, views, and stages, leveraging methods like contrastive learning and optimal transport theory; (2) a hard negative mining method that employs soft labels for multi-modality alignments and enforces the hard negative pair discrimination; and (3) a Gated Cross-Attention Module for Med-VQA that integrates the answer vocabulary as prior knowledge and selects relevant information from it. Our framework outperforms the previous state-of-the-art on widely used Med-VQA datasets like RAD-VQA, SLAKE, PathVQA and VQA-2019.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。