FedHQ通过混合量化提升联邦学习效率与精度,自动适配设备和数据差异。
FedHQ: Hybrid Runtime Quantization for Federated Learning
- 融合后训练量化与感知训练量化,动态选择最优策略。
- 训练速度最高提升2.47倍,精度最高提升11.15%,开销极低。
- 适合异构设备与数据分布的联邦学习场景,尤其关注效率与准确率平衡。
联邦学习(FL)是一种保护数据隐私的分布式训练方法,但效率较低。量化作为有效的训练优化技术,已被广泛研究用于集成到联邦学习中。然而,许多研究未考虑不同量化策略(如后训练量化PTQ或量化感知训练QAT)在性能上的差异。因此,现有联邦学习量化方法仅依赖单一策略,牺牲速度或精度。为高效加速联邦学习并保持跨多种设置下的分布式收敛精度,本文提出一种结合PTQ与QAT的混合量化方法。通过案例研究验证其有效性。针对设备与数据异构带来的速度与精度建模难题,提出硬件相关分析与数据分布相关分析,以识别策略选择的权衡边界。基于此,提出新型框架FedHQ,可自动为联邦学习系统分配最优混合策略。具体而言,FedHQ采用粗粒度全局初始化与细粒度基于机器学习的调整,确保效率与鲁棒性。实验表明,FedHQ实现最高2.47倍训练加速,精度提升达11.15%,额外开销可忽略不计。
原文摘要 · Abstract (English)
Federated Learning (FL) is a decentralized model training approach that preserves data privacy but struggles with low efficiency. Quantization, a powerful training optimization technique, has been widely explored for integration into FL. However, many studies fail to consider the distinct performance attribution between particular quantization strategies, such as post-training quantization (PTQ) or quantization-aware training (QAT). As a result, existing FL quantization methods rely solely on either PTQ or QAT, optimizing for speed or accuracy while compromising the other. To efficiently accelerate FL and maintain distributed convergence accuracy across various FL settings, this paper proposes a hybrid quantitation approach combining PTQ and QAT for FL systems. We conduct case studies to validate the effectiveness of using hybrid quantization in FL. To solve the difficulty of modeling speed and accuracy caused by device and data heterogeneity, we propose a hardware-related analysis and data-distribution-related analysis to help identify the trade-off boundaries for strategy selection. Based on these, we proposed a novel framework named FedHQ to automatically adopt optimal hybrid strategy allocation for FL systems. Specifically, FedHQ develops a coarse-grained global initialization and fine-grained ML-based adjustment to ensure efficiency and robustness. Experiments show that FedHQ achieves up to 2.47x times training acceleration and up to 11.15% accuracy improvement and negligible extra overhead.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。