用专家投票与混合模型提升医疗多模态哈希检索速度与精度
Enhancing Medical Cross-Modal Hashing Retrieval using Dropout-Voting Mixture-of-Experts Fusion
- 引入丢弃投票与专家混合机制融合多模态特征
- 在低内存环境下实现高精度与快速检索
- 适合需要高效医疗图像文本检索的临床与研究场景
近年来，基于图像与文本的跨模态检索在医疗领域日益活跃。医学数据模态丰富，推动了跨模态检索在图像解读、数据驱动诊断支持和医学教育中的重要性。随着医疗机构间分布式数据整合以提升互操作性，检索系统需兼顾速度、内存效率与准确率，应对现代医疗实践中的数据量激增。本文提出一种新框架，在CLIP基础上集成丢弃投票与基于专家混合（MoE）的对比融合模块，并引入混合损失函数，构建名为MCMFH的医疗跨模态融合哈希检索模型。实验在放射科与非放射科医疗数据集上验证，证明该方法可在低内存环境中同时实现高精度与快速检索。
原文摘要 · Abstract (English)
In recent years, cross-modal retrieval using images and text has become an active area of research, especially in the medical domain. The abundance of data in various modalities in this field has led to a growing importance of cross-modal retrieval for efficient image interpretation, data-driven diagnostic support, and medical education. In the context of the increasing integration of distributed medical data across healthcare facilities with the objective of enhancing interoperability, it is imperative to optimize the performance of retrieval systems in terms of the speed, memory efficiency, and accuracy of the retrieved data. This necessity arises in response to the substantial surge in data volume that characterizes contemporary medical practices. In this study, we propose a novel framework that incorporates dropout voting and mixture-of-experts (MoE) based contrastive fusion modules into a CLIP-based cross-modal hashing retrieval structure. We also propose the application of hybrid loss. So we now call our model MCMFH which is a medical cross-modal fusion hashing retrieval. Our method enables the simultaneous achievement of high accuracy and fast retrieval speed in low-memory environments. The model is demonstrated through experiments on radiological and non-radiological medical datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。