提出StaRFM框架，提升视觉语言与医学图像模型在分布偏移下的可靠性与置信度准确性。
Calibrated and Robust Foundation Models for Vision-Language and Medical Image Tasks Under Distribution Shift
- 融合费雪信息惩罚与置信度错配惩罚，分别缓解嵌入偏移与分割不确定性问题。
- 在19个视觉数据集上提升3.5%准确率，医疗任务中分割评分提高4.2%，误差降低28%。
- 适用于跨中心医学影像、低样本迁移场景，可直接接入现有模型使用。
CLIP和SAM等基础模型通过少样本迁移学习推动了计算机视觉与医学影像发展，助力药物发现（CADD）在数据有限时的应用。然而其部署面临两大挑战：分布偏移（如不同中心图像采集差异导致的前后数据分布不一致）与置信度错配（引发过度自信错误）。视觉语言模型（如CLIP）受2D嵌入偏移影响，而医学模型（如SAM）则面临3D域偏移（如扫描仪差异）及体素级校准需求。现有方法多为领域特定。本文提出StaRFM，结合费雪信息惩罚（FIP）与置信度错配惩罚（CMP），统一应对两类问题。FIP通过局部块正则化扩展至3D，减少嵌入偏移；CMP重构为体素级预测，优化分割不确定性。理论推导基于PAC-Bayes边界：FIP控制泛化能力（通过Fisher-Rao范数），CMP最小化Brier得分以降低校准误差。实验显示，StaRFM在19个视觉数据集（如ImageNet、Office-Home）上提升3.5%准确率，且ECE降低28%；在医疗基准（如BraTS、ATLAS）上，分割评分（DSC）提升4.2%，豪斯多夫距离（HD95）改善4.8mm，跨域差距缩小最高达20%。该框架即插即用，仅需少量架构改动。代码与模型已公开于https://anonymous.4open.science/r/StaRFM-C0CD/
原文摘要 · Abstract (English)
Foundation models like CLIP and SAM have advanced computer vision and medical imaging via low-shot transfer learning, aiding CADD with limited data. However, their deployment faces two key challenges. \textit{distribution shift} where pre-training and post-training data distributions differ (e.g., due to inter-center image acquisition) and \textit{confidence misalignment}, which leads to overconfident errors. These issues surface differently, vision-language models (e.g., CLIP) suffer from 2D embedding shift (image-text misalignment), while medical models (e.g., SAM) encounter 3D domain shifts (e.g., scanner variation) and voxel-wise calibration need. Existing solutions are domain-specific. We propose \textbf{StaRFM}, a fusion of Fisher information penalty (FIP) and confidence misalignment penalty (CMP) tackling both challenges. It applies FIP, extended to 3D via patch-wise regularization, to reduce embedding shift, and CMP, reformulated for voxel-level predictions, to calibrate segmentation uncertainty. We derive PAC-Bayes bounds. FIP controls generalization via the Fisher-Rao norm, and CMP reduces calibration error via Brier score minimization. StaRFM surpasses baselines by \texttt{+}3.5\% accuracy and 28\% lower ECE on 19 vision datasets (e.g., ImageNet, Office-Home), achieves +4.2\% DSC over SAM-FT and 4.8mm HD95 on medical benchmarks (e.g., BraTS, ATLAS), and reduces cross-domain gaps by up to 20\%. The framework is plug-and-play, requiring minimal architectural changes. Code and models are available at: \href{https://anonymous.4open.science/r/StaRFM-C0CD/}{\textcolor{blue}{\underline{StaRFM}}}
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。