提出CYCle协议,让去中心化学习中各方合作更公平高效
CYCle: Choosing Your Collaborators Wisely to Enhance Collaborative Fairness in Decentralized Learning
- 基于梯度对齐设计信誉评分,动态选择合作对象
- 同时最大化平均收益、最小化收益差异,确保无损且公平
- 适用于数据高度异构场景,适合关注公平性的分布式学习研究者
协同学习(CL)允许多方在不共享原始数据的前提下联合训练模型。现有方法多依赖中心化协调,仅关注提升平均准确率,忽视了收益分配的公平性。本文指出,传统基于准确率相关性的公平性度量忽略了负向协作收益的问题,主张同时最大化平均协作增益(MCG)和最小化协作增益离散度(CGS)更为合理。为此,我们提出CYCle协议,通过本地交叉熵与蒸馏损失间的梯度对齐计算信誉分,在私有去中心化学习框架中实现该目标。进一步将协议扩展至基于传闻的算法(如Gossip-SGD)。理论分析表明,在高异构性的双客户端均值估计任务中,CYCle优于标准FedAvg。实验验证了其在数据分布高度偏斜情况下仍能保障所有参与方获得正向且公平的协作收益。
原文摘要 · Abstract (English)
Collaborative learning (CL) enables multiple participants to jointly train machine learning (ML) models on decentralized data sources without raw data sharing. While the primary goal of CL is to maximize the expected accuracy gain for each participant, it is also important to ensure that the gains are fairly distributed: no client should be negatively impacted, and gains should reflect contributions. Most existing CL methods require central coordination and focus only on gain maximization, overlooking fairness. In this work, we first show that the existing measure of collaborative fairness based on the correlation between accuracy values without and with collaboration has drawbacks because it does not account for negative collaboration gain. We argue that maximizing mean collaboration gain (MCG) while simultaneously minimizing the collaboration gain spread (CGS) is a fairer alternative. Next, we propose the CYCle protocol that enables individual participants in a private decentralized learning (PDL) framework to achieve this objective through a novel reputation scoring method based on gradient alignment between the local cross-entropy and distillation losses. We further extend the CYCle protocol to operate on top of gossip-based decentralized algorithms such as Gossip-SGD. We also theoretically show that CYCle performs better than standard FedAvg in a two-client mean estimation setting under high heterogeneity. Empirical experiments demonstrate the effectiveness of the CYCle protocol to ensure positive and fair collaboration gain for all participants, even in cases where the data distributions of participants are highly skewed.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。