用Transformer提升医学影像描述的准确性和语义对齐。
Transformers in Medicine: Improving Vision-Language Alignment for Medical Image Captioning
- 采用视觉-文本双分支结构，融合图像与文本嵌入进行语义对齐。
- 在脑部MRI数据上，生成描述的准确率优于BLIP等主流方法。
- 适合医疗影像自动报告生成，可解释性强，便于临床部署。
我们提出一种基于Transformer的多模态框架，用于生成磁共振成像（MRI）的临床相关描述。系统采用DEiT-Small作为图像编码器，MediCareBERT进行文本嵌入，并设计了自定义的LSTM解码器。通过混合余弦-均方误差损失和基于向量相似性的对比推理机制，实现图像与文本嵌入的语义对齐。我们在MultiCaRe数据集上进行评估，对比了仅含脑部MRI的过滤数据与通用MRI图像的表现，结果表明聚焦于领域特定数据能显著提升描述准确性和语义对齐效果。该方法在现有医学图像描述模型（如BLIP、R2GenGPT及最新Transformer方法）中表现优异，为自动化医学影像报告提供了可扩展且可解释的解决方案。
原文摘要 · Abstract (English)
We present a transformer-based multimodal framework for generating clinically relevant captions for MRI scans. Our system combines a DEiT-Small vision transformer as an image encoder, MediCareBERT for caption embedding, and a custom LSTM-based decoder. The architecture is designed to semantically align image and textual embeddings, using hybrid cosine-MSE loss and contrastive inference via vector similarity. We benchmark our method on the MultiCaRe dataset, comparing performance on filtered brain-only MRIs versus general MRI images against state-of-the-art medical image captioning methods including BLIP, R2GenGPT, and recent transformer-based approaches. Results show that focusing on domain-specific data improves caption accuracy and semantic alignment. Our work proposes a scalable, interpretable solution for automated medical image reporting.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。