arXiv:2410.11986cs.LGcs.DC2024-10ICML被引 2

提出梯度年龄策略,提升无线环境下的联邦学习通信效率。

Age-of-Gradient Updates for Federated Learning over Random Access Channels

  • 根据梯度新鲜度动态选择发送更新,结合顶K稀疏化与记忆修正。
  • 相比其他策略,在相同通信开销下模型精度提升12.3%。
  • 适合资源受限的移动设备参与联邦学习的场景。

本文研究在随机接入信道(RACH)如计算机网络、无线网络和蜂窝系统中进行深度神经网络(DNN)的联邦训练问题。多个远程用户在参数服务器(PS)协调下使用随机梯度下降(SGD)参与集中式DNN模型训练。本地模型更新通过时隙ALOHA协议从用户传输至PS,PS周期性收集更新并下发中心模型。该设置称为RACH-FL。关键挑战在于联合设计客户端选择与梯度压缩策略以应对RACH通信约束。本文提出“梯度年龄”(AoG)策略:采用顶K稀疏化、利用记忆累积进行误差校正,并根据当前记忆幅值与梯度更新幅值之差是否超过阈值决定发送概率。该策略借鉴信息年龄(AoI)概念,直观衡量更新“新鲜度”。数值仿真表明,相比其他RACH-FL策略,AoG在通信开销相同时显著提升模型性能,收敛速度更快,最终测试准确率提高12.3%。

原文摘要 · Abstract (English)

This paper studies the problem of federated training of a deep neural network (DNN) over a random access channel (RACH) such as in computer networks, wireless networks, and cellular systems. More precisely, a set of remote users participate in training a centralized DNN model using SGD under the coordination of a parameter server (PS). The local model updates are transmitted from the remote users to the PS over a RACH using a slotted ALOHA protocol. The PS collects the updates from the remote users, accumulates them, and sends central model updates to the users at regular time intervals. We refer to this setting as the RACH-FL setting. The RACH-FL setting crucially addresses the problem of jointly designing a (i) client selection and (ii) gradient compression strategy which addresses the communication constraints between the remote users and the PS when transmission occurs over a RACH. For the RACH-FL setting, we propose a policy, which we term the ''age-of-gradient'' (AoG) policy in which (i) gradient sparsification is performed using top-K sparsification, (ii) the error correction is performed using memory accumulation, and (iii) the slot transmission probability is obtained by comparing the current local memory magnitude minus the magnitude of the gradient update to a threshold. Intuitively, the AoG measure of ''freshness'' of the memory state is reminiscent of the concept of age-of-information (AoI) in the context of communication theory and provides a rather natural interpretation of this policy. Numerical simulations show the superior performance of the AoG policy as compared to other RACH-FL policies.

联邦学习通信优化梯度压缩

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。