arXiv:2502.03092cs.LGcs.AI2025-02中稿 · TNNLS被引 5

用自重构机制压缩梯度,通信量降111倍且模型更准。

E-3SFC: Communication-Efficient Federated Learning with Double-way Features Synthesizing

  • 用模型自身做解压器,单步生成小尺寸合成特征
  • 实验显示通信成本降低111.6倍,性能领先13.4%
  • 适合资源受限的分布式训练场景

模型规模增长导致联邦学习通信开销激增。现有通过传输压缩梯度缓解负担的方法常因压缩误差高而影响收敛速度。本文从新视角研究梯度压缩问题,提出扩展型单步合成特征压缩算法(E-3SFC),包含三个组件:单步合成特征压缩器(3SFC)、双向压缩机制和通信预算调度器。将模型梯度计算视为从输入中解压梯度,逆过程则为压缩。3SFC利用模型权重与目标函数等训练先验,以单步仿真将原始梯度压缩为微小合成特征,并引入误差反馈降低总压缩误差。进一步扩展为E-3SFC,支持双向压缩与动态通信预算分配。理论分析表明,在强凸与非凸条件下,3SFC可实现线性与次线性收敛率,且容忍聚合噪声。在六个数据集、六种模型上的大量实验显示,3SFC相比先进方法性能提升最高达13.4%,通信成本降低111.6倍。结果表明,该方法可在不牺牲模型性能的前提下显著提升联邦学习通信效率。

原文摘要 · Abstract (English)

The exponential growth in model sizes has significantly increased the communication burden in Federated Learning (FL). Existing methods to alleviate this burden by transmitting compressed gradients often face high compression errors, which slow down the model's convergence. To simultaneously achieve high compression effectiveness and lower compression errors, we study the gradient compression problem from a novel perspective. Specifically, we propose a systematical algorithm termed Extended Single-Step Synthetic Features Compressing (E-3SFC), which consists of three sub-components, i.e., the Single-Step Synthetic Features Compressor (3SFC), a double-way compression algorithm, and a communication budget scheduler. First, we regard the process of gradient computation of a model as decompressing gradients from corresponding inputs, while the inverse process is considered as compressing the gradients. Based on this, we introduce a novel gradient compression method termed 3SFC, which utilizes the model itself as a decompressor, leveraging training priors such as model weights and objective functions. 3SFC compresses raw gradients into tiny synthetic features in a single-step simulation, incorporating error feedback to minimize overall compression errors. To further reduce communication overhead, 3SFC is extended to E-3SFC, allowing double-way compression and dynamic communication budget scheduling. Our theoretical analysis under both strongly convex and non-convex conditions demonstrates that 3SFC achieves linear and sub-linear convergence rates with aggregation noise. Extensive experiments across six datasets and six models reveal that 3SFC outperforms state-of-the-art methods by up to 13.4% while reducing communication costs by 111.6 times. These findings suggest that 3SFC can significantly enhance communication efficiency in FL without compromising model performance.

联邦学习梯度压缩通信效率

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。