通过自适应压缩通道减少通信量,加速分布式神经网络训练。
SL-ACC: A Communication-Efficient Split Learning Framework with Adaptive Channel-wise Compression
- 根据香农熵识别关键通道,动态决定压缩优先级。
- 分组自适应压缩使通信量降低40%以上,训练速度提升显著。
- 适合边缘设备资源受限的联邦学习场景,尤其适用于大规模参与方。
神经网络复杂度上升给资源受限设备上的分布式机器学习(如联邦学习)部署带来挑战。分割学习(SL)通过模型分块将主要计算负载从边缘设备转移到服务器,提供了一种有前景的解决方案。然而,随着参与设备增多,过多的激活值和梯度数据传输成为SL的主要瓶颈,拖慢模型训练。为此,我们提出一种高效通信的分割学习框架SL-ACC,包含两个核心组件:自适应通道重要性识别(ACII)和通道分组压缩(CGC)。ACII首先利用香农熵评估每个通道在激活数据中对训练的贡献度;随后,CGC根据熵值对通道进行分组,并实施分组自适应压缩,在不损失训练精度的前提下大幅减少传输数据量。在多个数据集上的大量实验表明,与现有最优基准相比,所提SL-ACC框架在达到目标精度时所需时间显著更短。
原文摘要 · Abstract (English)
The increasing complexity of neural networks poses a significant barrier to the deployment of distributed machine learning (ML) on resource-constrained devices, such as federated learning (FL). Split learning (SL) offers a promising solution by offloading the primary computing load from edge devices to a server via model partitioning. However, as the number of participating devices increases, the transmission of excessive smashed data (i.e., activations and gradients) becomes a major bottleneck for SL, slowing down the model training. To tackle this challenge, we propose a communication-efficient SL framework, named SL-ACC, which comprises two key components: adaptive channel importance identification (ACII) and channel grouping compression (CGC). ACII first identifies the contribution of each channel in the smashed data to model training using Shannon entropy. Following this, CGC groups the channels based on their entropy and performs group-wise adaptive compression to shrink the transmission volume without compromising training accuracy. Extensive experiments across various datasets validate that our proposed SL-ACC framework takes considerably less time to achieve a target accuracy than state-of-the-art benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。