联邦学习让CLIP模型预测变不准，新方法能自动提升可靠性。
How (Mis)calibrated is Your Federated CLIP and What To Do About It?
- 用LoRA微调特定模块，比整体微调更利于保持预测可信度。
- 实验显示新方法在多个数据集上显著改善校准性能，减少对额外校准的需求。
- 适合关注联邦学习中模型可信度的研究者和应用开发者。
尽管视觉语言模型如CLIP已广泛研究，但其校准性——影响预测可靠性的重要因素——却未受足够关注。现有工作仅探讨了离线场景下的CLIP校准，而联邦学习（FL）环境下微调CLIP的影响尚不清楚。本文分析了文本提示调优方法在联邦学习中的表现，发现其会恶化校准指标；同时评估了四种全局聚合方式下现有的训练中校准技术，结果表明改进有限。我们的研究揭示，问题根源不仅在于聚合或校准策略，更在于微调的组件选择。为此，我们提出$ ext{FL}^2 ext{oRA}$，一种基于LoRA的简单方法，在联邦学习中自然提升校准性，并分析其有效性的关键因素。多基准测试表明，$ ext{FL}^2 ext{oRA}$能持续生成校准良好的模型，降低对显式校准流程的依赖。代码已开源。
原文摘要 · Abstract (English)
While vision-language models like CLIP have been extensively studied, their calibration, crucial for reliable predictions, has received limited attention. Although a few prior works have examined CLIP calibration in offline settings, the impact of fine-tuning CLIP in a federated learning (FL) setup remains unexplored. In this work, we investigate how FL affects CLIP calibration and propose strategies to improve reliability in this distributed setting. We first analyze Textual Prompt Tuning approaches and show that they degrade calibration metrics when operating under FL. We also evaluate existing in-training calibration techniques across four global aggregation methods, finding that they provide limited improvements. Our results suggest that the key challenge lies not only in how we aggregate or calibrate, but in which components we choose to fine-tune. Motivated by this insight, we propose $\text{FL}^2\text{oRA}$, a straightforward LoRA-based approach that naturally improves calibration in FL, and we analyze the factors behind its effectiveness. Experiments on multiple benchmarks demonstrate that $\text{FL}^2\text{oRA}$ consistently produces well-calibrated models, reducing the need for explicit calibration procedures. Codes are available at https://github.com/mainaksingha01/FL2oRA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。