用预训练大模型层提升医学图像分割精度
MedVisionLlama: Leveraging Pre-Trained Large Language Model Layers to Enhance Medical Image Segmentation
- 将冻结的LLM Transformer块融入ViT编码器，增强特征提取
- 平均Dice分数从0.74提升至0.79，其他指标同步改善
- 适合关注医学图像分析与Transformer融合的科研人员
大型语言模型（LLMs）在文本数据中表现出色，正被探索用于提升医学图像分割性能。本文通过在基于视觉变换器（ViT）的模型编码器中引入预训练的LLM Transformer块，显著提升了多种医学影像模态的分割效果。提出了一种混合注意力机制，结合全局与局部特征学习，并设计多尺度融合模块以聚合不同层级特征。实验显示，该方法使平均Dice分数从0.74提升至0.79，同时准确率、精确率和Jaccard指数均获得提升。结果表明，基于LLM的Transformer结构能有效优化医学图像分割，显著增强模型精度与鲁棒性。源代码与实现已公开于：https://github.com/AS-Lab/Marthi-et-al-2025-MedVisionLlama-Pre-Trained-LLM-Layers-to-Enhance-Medical-Image-Segmentation
原文摘要 · Abstract (English)
Large Language Models (LLMs), known for their versatility in textual data, are increasingly being explored for their potential to enhance medical image segmentation, a crucial task for accurate diagnostic imaging. This study explores enhancing Vision Transformers (ViTs) for medical image segmentation by integrating pre-trained LLM transformer blocks. Our approach, which incorporates a frozen LLM transformer block into the encoder of a ViT-based model, leads to substantial improvements in segmentation performance across various medical imaging modalities. We propose a Hybrid Attention Mechanism that combines global and local feature learning with a Multi-Scale Fusion Block for aggregating features across different scales. The enhanced model shows significant performance gains, including an average Dice score increase from 0.74 to 0.79 and improvements in accuracy, precision, and the Jaccard Index. These results demonstrate the effectiveness of LLM-based transformers in refining medical image segmentation, highlighting their potential to significantly boost model accuracy and robustness. The source code and our implementation are available at: https://github.com/AS-Lab/Marthi-et-al-2025-MedVisionLlama-Pre-Trained-LLM-Layers-to-Enhance-Medical-Image-Segmentation
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。