通过参数压缩降低联邦学习通信开销,精度损失小。
Trading-off Accuracy and Communication Cost in Federated Learning
- 用可训练参数向量压缩模型权重,减少传输数据量。
- 通信成本降低34倍,优于现有最优方法多个数量级。
- 适合资源受限设备参与大规模联邦学习的场景。
基于周等人提出的剪枝训练范式,我们设计了一种联邦学习协议,实现了34倍的通信成本降低。我们的框架通过将网络权重 $m{w}$ 编码为可训练参数 $m{p}$,即 $m{w} = Q m{p}$,其中 $Q$ 是一个预生成的稀疏随机矩阵(训练中固定)。相较于之前工作(当 $Q$ 为对角阵且 $m{p}$ 维度与 $m{w}$ 相同时),我们发现 $m{p}$ 可远小于 $m{w}$,只需适当降低 $Q$ 的稀疏性即可保持相同精度。由于服务器与客户端仅需交换 $m{p}$,通信开销显著下降。此外,我们提供了理论分析,建立了训练采样与随机凸几何之间的新联系。
原文摘要 · Abstract (English)
Leveraging the training-by-pruning paradigm introduced by Zhou et al. and Isik et al. introduced a federated learning protocol that achieves a 34-fold reduction in communication cost. We achieve a compression improvements of orders of orders of magnitude over the state-of-the-art. The central idea of our framework is to encode the network weights $\vec w$ by a the vector of trainable parameters $\vec p$, such that $\vec w = Q\cdot \vec p$ where $Q$ is a carefully-generate sparse random matrix (that remains fixed throughout training). In such framework, the previous work of Zhou et al. [NeurIPS'19] is retrieved when $Q$ is diagonal and $\vec p$ has the same dimension of $\vec w$. We instead show that $\vec p$ can effectively be chosen much smaller than $\vec w$, while retaining the same accuracy at the price of a decrease of the sparsity of $Q$. Since server and clients only need to share $\vec p$, such a trade-off leads to a substantial improvement in communication cost. Moreover, we provide theoretical insight into our framework and establish a novel link between training-by-sampling and random convex geometry.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。