分离医学影像中的临床信号与无关噪声，提升模型在真实场景下的可靠性。
Decoupling Clinical and Class-Agnostic Features for Reliable Few-Shot Adaptation under Shift
- 用LoRA和可学习提示词解耦临床相关特征与无关噪声。
- 在分布外数据上提升11.4%准确率和3.3%宏平均F1值。
- 适合需要高可靠性的医疗视觉语言模型部署场景。
医学视觉语言模型（VLMs）在临床决策支持中具有潜力，但在分布偏移下仍面临可靠性挑战。由于成像协议和自由文本报告的多样性，模型常学习到任务无关的相关性，限制其泛化能力并增加实际应用中的失败风险。我们提出DRiFt框架，通过参数高效微调（LoRA）和可学习提示词，显式分离临床相关信号与任务无关噪声。为增强跨模态对齐并降低不确定性，我们基于多样化医学数据集生成高质量、临床相关的图文配对。该方法在分布内性能上比以往提示方法提升11.4%的Top-1准确率和3.3%的宏平均F1，同时在未见数据集上保持强鲁棒性。消融实验证明，解耦任务相关特征并优化对齐显著提升模型泛化能力，减少域偏移下的不可预测行为。这些发现有助于构建更安全、可信的临床用视觉语言模型。代码已公开于https://github.com/rumaima/DRiFt。
原文摘要 · Abstract (English)
Medical vision-language models (VLMs) offer promise for clinical decision support, yet their reliability under distribution shifts remains a major concern for safe deployment. These models often learn task-agnostic correlations due to variability in imaging protocols and free-text reports, limiting their generalizability and increasing the risk of failure in real-world settings. We propose DRiFt, a structured feature decoupling framework that explicitly separates clinically relevant signals from task-agnostic noise using parameter-efficient tuning (LoRA) and learnable prompt tokens. To enhance cross-modal alignment and reduce uncertainty, we curate high-quality, clinically grounded image-text pairs by generating captions for a diverse medical dataset. Our approach improves in-distribution performance by +11.4% Top-1 accuracy and +3.3% Macro-F1 over prior prompt-based methods, while maintaining strong robustness across unseen datasets. Ablation studies reveal that disentangling task-relevant features and careful alignment significantly enhance model generalization and reduce unpredictable behavior under domain shift. These insights contribute toward building safer, more trustworthy VLMs for clinical use. The code is available at https://github.com/rumaima/DRiFt.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。