轻量级框架让通用视觉语言模型高效适配胸部X光诊断。
Adapting Foundation Vision-Language Models to Medical Diagnosis via Query-Driven Expert Bridging
- 用可学习查询令牌实现多视角高分辨率采样与领域对齐。
- 在五项胸部放射学任务中提升6-15%的AUC,优于现有方法。
- 兼容8种主流模型,适合快速部署于各类视觉语言模型。
视觉语言基础模型在自然图像分类中表现优异,但直接用于医学影像时受限于严重的领域差异、分辨率不匹配及临床诊断的多标签特性。从头训练专用医学基础模型成本高且数据需求大。本文提出MedBridge,一种轻量级适配框架,通过联合域对齐、分辨率保持和多标签推理,利用互补的VLM专家实现领域差距缓解。MedBridge将预训练VLM转为多视图查询编码器,在中间层注入一组可学习查询令牌,实现非破坏性域对齐并保留细微病理线索;这些查询令牌作为路由信号,驱动专家混合模型动态整合异构基础模型,无需共享表示空间。在五个胸部X光基准上评估,该方法在跨域泛化(分布外迁移)和同域专精(同分布调优)中均表现更优,多标签胸腔疾病诊断的AUC提升6-15%。此外,MedBridge具备模型无关性,可在八种不同VLM(如CLIP、LLaVA、Qwen-VL、MedGemma)上灵活扩展,证明其可将任意基础模型转化为强大医疗诊断工具。代码将在接受后发布。
原文摘要 · Abstract (English)
Vision-language foundation models achieve promising performance in natural image classification, yet their direct application to medical imaging is limited by severe domain shifts, resolution mismatches, and the multi-label nature of clinical diagnosis. Training dedicated medical foundation models from scratch, however, is costly and data-intensive. Here, we propose MedBridge, a lightweight adaptation framework that opens a new direction in domain-gap mitigation by jointly combining domain alignment, resolution preservation, and multi-label reasoning via complementary VLM experts for medical image diagnosis. Specifically, MedBridge transforms pretrained VLMs into multi-view query encoders that inject a compact set of learnable query tokens into intermediate layers, enabling non-destructive domain alignment while preserving fine-grained pathological cues via multi-view high-resolution sampling. These query tokens further act as routing signals for a mixture-of-experts, dynamically integrating heterogeneous foundation models for multi-label reasoning without requiring a shared representation space. We evaluated MedBridge on five chest radiograph benchmarks in three key adaptation tasks. MedBridge demonstrates superior performance in both cross-domain generalization (out-of-distribution transfer) and in-domain specialization (same-distribution tuning) settings, yielding a significant 6-15% AUC improvement over state-of-the-art adaptation methods for multi-label thoracic disease diagnosis. Furthermore, MedBridge is model-agnostic and demonstrates broad extensibility across eight diverse VLMs (e.g., CLIP, LLaVA, Qwen-VL, MedGemma), highlighting its ability to flexibly adapt arbitrary foundation models into a powerful medical diagnostic tool. Our code will be released upon acceptance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。