提出高效医学图文问答融合模型,兼顾性能与计算效率。
Efficient Bilinear Attention-based Fusion for Medical Visual Question Answering
- 采用正交性约束+多头注意力+双线性注意力网络融合模态信息
- 参数量仅为Transformer的1/3,浮点运算量降低至1/4,性能相当
- 适合资源受限的临床场景,尤其适用于闭合式问题回答
医学视觉问答(MedVQA)在医疗图像理解与自然语言处理交汇处日益受到关注,有望辅助诊断决策并减轻放射科等领域的负担。尽管现有方法多依赖统一的大规模预训练视觉-语言模型,但该领域对更高效融合机制的研究仍较有限。本文提出一种名为OMniBAN的融合模型,整合正交性损失、多头注意力与双线性注意力网络,在保证良好性能的同时实现高计算效率。大量实验表明,双线性注意力融合可逼近跨模态Transformer等大型模型的性能。结果表明,OMniBAN仅需约2/3的参数量(相比基于Transformer的协同注意力)和约1/4的浮点运算量(FLOPs),在两个关键MedVQA基准上达到相当甚至略优的整体表现,尤其在闭合式问题上表现突出。该平衡效率与精度的设计,使其成为实际医疗图像问答中计算资源受限场景下的可行选择。
原文摘要 · Abstract (English)
Medical Visual Question Answering (MedVQA) has attracted growing interest at the intersection of medical image understanding and natural language processing for clinical applications. By interpreting medical images and providing precise answers to relevant clinical inquiries, MedVQA has the potential to support diagnostic decision-making and reduce workload across various fields like radiology. While recent approaches rely heavily on unified large pre-trained Visual-Language Models, research on more efficient fusion mechanisms remains relatively limited in this domain. In this paper, we introduce a fusion model, OMniBAN, that integrates Orthogonality loss, Multi-head attention, and a Bilinear Attention Network to achieve high computational efficiency as well as solid performance. We conduct comprehensive experiments and demonstrate how bilinear attention fusion can approximate the performance of larger fusion models like cross-modal Transformer. Our results show that OMniBAN requires fewer parameters (approximately 2/3 of Transformer-based Co-Attention) and substantially lower FLOPs (approximately 1/4), while achieving comparable overall performance and even slight improvements on closed-ended questions on two key MedVQA benchmarks. This balance between efficiency and accuracy suggests that OMniBAN could be a viable option for real-world medical image question answering, where computational resources are often constrained.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。