仅用类别均值即可估算协方差,实现零训练联邦学习的高效分类初始化。
Covariances for Free: Exploiting Mean Distributions for Training-free Federated Learning
- 基于客户端上传的类别均值,无偏估计类间协方差矩阵
- 性能提升4%-26%,通信开销仅为共享二阶统计量的极小部分
- 适合资源受限场景,可作为后续微调的优越初始化方案
使用预训练模型可缓解数据异构性并加速联邦学习。近期研究探索了无需训练的方法,利用一阶与二阶统计量在服务器端聚合本地客户端数据分布,实现高性能而无需训练。本文提出一种训练自由方法,仅依赖客户端上传的类别均值(一阶统计量),通过无偏估计获得类协方差矩阵,并用于初始化全局分类器,从而在不共享协方差的前提下利用其信息。我们证明仅使用类内协方差即可获得更优的分类器初始化。相比仅共享类别均值的方法,性能提升4%-26%,且通信成本完全相同;相较于共享二阶统计量的方法,性能相当或更优,但通信开销大幅降低。该方法比联邦提示调优更高效,同时表现更优。将本方法用于初始化分类器后,再进行联邦微调或线性探测,仍能获得更高性能。代码已开源:https://github.com/dipamgoswami/FedCOF。
原文摘要 · Abstract (English)
Using pre-trained models has been found to reduce the effect of data heterogeneity and speed up federated learning algorithms. Recent works have explored training-free methods using first- and second-order statistics to aggregate local client data distributions at the server and achieve high performance without any training. In this work, we propose a training-free method based on an unbiased estimator of class covariance matrices which only uses first-order statistics in the form of class means communicated by clients to the server. We show how these estimated class covariances can be used to initialize the global classifier, thus exploiting the covariances without actually sharing them. We also show that using only within-class covariances results in a better classifier initialization. Our approach improves performance in the range of 4-26% with exactly the same communication cost when compared to methods sharing only class means and achieves performance competitive or superior to methods sharing second-order statistics with dramatically less communication overhead. The proposed method is much more communication-efficient than federated prompt-tuning methods and still outperforms them. Finally, using our method to initialize classifiers and then performing federated fine-tuning or linear probing again yields better performance. Code is available at https://github.com/dipamgoswami/FedCOF.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。