用大模型指导医疗视觉模型预训练，轻量高效且泛化强。
VIVID-Med: LLM-Supervised Structured Pretraining for Deployable Medical ViTs
- 用冻结大模型生成结构化标签，精准捕捉临床发现语义关系。
- 仅用500倍少数据实现85.88%的宏观AUC，超越现有方法6.65点。
- 训练后移除大模型，得轻量可部署的视觉变压器，适合临床落地。
视觉-语言预训练推动了医学图像分析的发展。但现有方法多采用独热标签或自由文本监督视觉编码器，难以有效捕捉临床发现间的复杂语义关系。本文提出VIVID-Med框架，利用冻结的大语言模型（LLM）作为结构化语义教师，预训练医疗视觉变换器（ViTs）。该方法通过统一医学模式（UMS）将临床发现转化为可验证的JSON字段-状态对，并采用答案感知掩码聚焦优化。进一步通过结构化预测分解（SPD）将交叉注意力划分为正交性正则化的查询组，提取互补视觉特征。关键的是，训练后丢弃LLM，得到轻量、可部署的纯视觉变压器主干。在多个场景评估中，于CheXpert线性探测达到0.8588宏观AUC，优于BiomedCLIP 6.65个百分点，且仅使用其1/500的数据；在零样本跨域迁移至NIH ChestX-ray14达0.7225宏观AUC；在跨模态任务中，于LIDC-IDRI肺结节分类达0.8413 AUC，OrganAMNIST 11器官分类达0.9969宏观AUC。VIVID-Med为临床场景提供高效、可扩展的轻量级视觉语言模型部署方案。
原文摘要 · Abstract (English)
Vision-language pretraining has driven significant progress in medical image analysis. However, current methods typically supervise visual encoders using one-hot labels or free-form text, neither of which effectively captures the complex semantic relationships among clinical findings. In this study, we introduce VIVID-Med, a novel framework that leverages a frozen large language model (LLM) as a structured semantic teacher to pretrain medical vision transformers (ViTs). VIVID-Med translates clinical findings into verifiable JSON field-state pairs via a Unified Medical Schema (UMS), utilizing answerability-aware masking to focus optimization. It then employs Structured Prediction Decomposition (SPD) to partition cross-attention into orthogonality-regularized query groups, extracting complementary visual aspects. Crucially, the LLM is discarded post-training, yielding a lightweight, deployable ViT-only backbone. We evaluated VIVID-Med across multiple settings: on CheXpert linear probing, it achieves a macro-AUC of 0.8588, outperforming BiomedCLIP by +6.65 points while using 500x less data. It also demonstrates robust zero-shot cross-domain transfer to NIH ChestX-ray14 (0.7225 macro-AUC) and strong cross-modality generalization to CT, achieving 0.8413 AUC on LIDC-IDRI lung nodule classification and 0.9969 macro-AUC on OrganAMNIST 11-organ classification. VIVID-Med offers a highly efficient, scalable alternative to deploying resource-heavy vision-language models in clinical settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。