针对联邦学习中数据异构与长尾分布难题,提出双提示机制提升小类识别能力。
CAPT: Class-Aware Prompt Tuning for Federated Long-Tailed Learning with Vision-Language Model
- 设计通用与类别感知双提示,兼顾全局与类别特异性知识
- 在多个长尾数据集上显著提升尾部类别准确率,整体性能不降
- 适用于医疗、工业等存在数据分布不均的联邦学习场景
在联邦学习中,非独立同分布数据与长尾分布的共存仍是关键挑战。尽管微调视觉语言模型(如CLIP)对缓解非独立同分布问题有潜力,但在联邦长尾场景下会导致尾部类别性能严重退化,且在强非独立同分布与长尾不平衡双重影响下,微调甚至无法带来改进。为此,我们提出类感知提示学习框架CAPT,利用预训练视觉语言模型有效应对数据异构性与长尾分布。CAPT引入双提示机制,融合通用提示与类别感知提示,使模型既能捕捉全局趋势,又能保留类别特异性信息。为更好实现跨客户端的知识聚合与共享,提出基于异质性的客户端聚类策略,依据数据分布分组,促进高效协作。在多种不同异构程度的长尾数据集上的大量实验表明,CAPT显著提升尾部类别性能,同时保持整体准确率,优于现有最先进方法。
原文摘要 · Abstract (English)
Effectively handling the co-occurrence of non-IID data and long-tailed distributions remains a critical challenge in federated learning. While fine-tuning vision-language models (VLMs) like CLIP has shown to be promising in addressing non-IID data challenges, this approach leads to severe degradation of tail classes in federated long-tailed scenarios. Under the composite effects of strong non-IID data distribution and long-tailed class imbalances, VLM fine-tuning may even fail to yield any improvement. To address this issue, we propose Class-Aware Prompt Learning for Federated Long-tailed Learning (CAPT), a novel framework that leverages a pre-trained VLM to effectively handle both data heterogeneity and long-tailed distributions. CAPT introduces a dual-prompt mechanism that synergizes general and class-aware prompts, enabling the framework to capture global trends while preserving class-specific knowledge. To better aggregate and share knowledge across clients, we introduce a heterogeneity-aware client clustering strategy that groups clients based on their data distributions, enabling efficient collaboration and knowledge sharing. Extensive experiments on various long-tailed datasets with different levels of data heterogeneity demonstrate that CAPT significantly improves tail class performance without compromising overall accuracy, outperforming state-of-the-art methods in federated long-tailed learning scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。