用预训练大模型提升医学图像分割，不改参数也能显著提效。
Pre-Trained LLM is a Semantic-Aware and Generalizable Segmentation Booster
- 将冻结的预训练大模型嵌入CNN分割框架，实现语义感知
- 在超声、皮肤镜等多模态数据上均提升分割性能
- 无需微调即可增强全局理解与局部建模能力
随着大语言模型在自然语言处理中的发展，本文发现：一个冻结的预训练大语言模型层可处理视觉令牌，用于医学图像分割任务。我们提出一种简单的混合结构，将预训练、冻结的大语言模型层集成到CNN编码器-解码器分割框架中（LLM4Seg）。令人惊讶的是，该设计在多种模态（包括超声、皮肤镜、肠镜和CT扫描）下，仅以极少量新增可训练参数，显著提升了分割性能。深入分析表明，大语言模型的语义感知能力可有效迁移至分割任务，增强了全局理解与局部建模能力。该效果在不同大模型（如LLaMA与DeepSeek）上均表现稳健。
原文摘要 · Abstract (English)
With the advancement of Large Language Model (LLM) for natural language processing, this paper presents an intriguing finding: a frozen pre-trained LLM layer can process visual tokens for medical image segmentation tasks. Specifically, we propose a simple hybrid structure that integrates a pre-trained, frozen LLM layer within the CNN encoder-decoder segmentation framework (LLM4Seg). Surprisingly, this design improves segmentation performance with a minimal increase in trainable parameters across various modalities, including ultrasound, dermoscopy, polypscopy, and CT scans. Our in-depth analysis reveals the potential of transferring LLM's semantic awareness to enhance segmentation tasks, offering both improved global understanding and better local modeling capabilities. The improvement proves robust across different LLMs, validated using LLaMA and DeepSeek.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。