用大模型嵌入提升医学图文对齐，让长报告理解更准
QwenCLIP: Boosting Medical Vision-Language Pretraining via LLM Embeddings and Prompt tuning
- 用Qwen3-Embedding替换原CLIP文本编码器，支持更长文本输入
- 在多个放射科数据集上表现优于现有方法，最高提升6.2%准确率
- 适合做医学影像+报告分析的研究者或临床辅助系统开发者
对比语言-图像预训练（CLIP）在计算机视觉和医学领域展现了强大泛化能力，但其文本编码器仅支持最多77个标记，难以表达长且信息丰富的放射科报告。近期使用领域专用编码器（如PubMedBERT或ClinicalBERT）的改进虽缓解了此问题，但仍受限于输入长度（通常为512标记）和较浅的语义理解能力。为此，我们提出QwenCLIP，将CLIP的文本编码器替换为基于大语言模型（LLM）的嵌入模块（如Qwen3-Embedding），并引入可学习提示（prompt tuning）以增强跨模态对齐。通过利用大模型的扩展上下文窗口与更丰富的表示能力，QwenCLIP能够从长篇临床文本中捕捉全面的医学语义，显著提升医学图像-文本对齐效果，并在多个放射科基准测试中取得优异性能。代码已公开于https://github.com/Wxy-24/QwenCLIP。
原文摘要 · Abstract (English)
Contrastive Language-Image Pretraining (CLIP) has demonstrated strong generalization for vision-language tasks in computer vision and medical domains, yet its text encoder accepts only up to 77 tokens, which limits its ability to represent long and information-rich radiology reports. Recent adaptations using domain-specific encoders, such as PubMedBERT or ClinicalBERT, mitigate this issue by leveraging medical corpora, but remain constrained by their limited input length (typically 512 tokens) and relatively shallow semantic understanding. To address these limitations, we propose QwenCLIP, a vision-language framework that replaces CLIP's text encoder with a large language model (LLM)-based embedding module (e.g., Qwen3-Embedding) and introduces learnable prompts to enhance cross-modal alignment. By leveraging the extended context window and richer representations of LLMs, QwenCLIP captures comprehensive medical semantics from long-form clinical text, substantially improving medical image-text alignment and downstream performance on radiology benchmarks. Our code is publicly available at https://github.com/Wxy-24/QwenCLIP.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。