通过固定门控结构提升联邦学习稳定性,减少训练轮次。
FedSQ: Optimized Weight Averaging via Fixed Gating
- 用预训练模型的结构副本生成固定二值门控掩码
- 仅优化量化副本并聚合,降低异构数据下的波动
- 适合跨组织迁移学习场景,尤其在数据非独立同分布时
联邦学习(FL)可在不共享原始数据的情况下实现多机构协作训练,但受限于统计异质性(客户端数据非独立同分布)和客户端漂移导致的简单权重平均不稳定。在多数跨孤岛部署中,FL通常从强预训练主干网络(如ImageNet-1K)开始,再适应本地域。受近期研究表明:类似ReLU的门控机制(结构知识)比其余参数值(量化知识)更早稳定启发,本文提出FedSQ(联邦结构-量化学习),一种基于双副本、分段线性视角的迁移初始化联邦方法。FedSQ冻结预训练模型的结构副本,以在联邦微调过程中生成固定二值门控掩码,仅对量化副本进行本地优化并在各轮间聚合。固定门控使学习退化为区间内仿射修正,显著提升了异构划分下的聚合稳定性。在两种卷积神经网络主干下,分别采用i.i.d.与Dirichlet划分的实验表明,FedSQ提升了鲁棒性,可减少达到最佳验证性能所需的训练轮数,同时在迁移设置中保持精度。
原文摘要 · Abstract (English)
Federated learning (FL) enables collaborative training across organizations without sharing raw data, but it is hindered by statistical heterogeneity (non-i.i.d.\ client data) and by instability of naive weight averaging under client drift. In many cross-silo deployments, FL is warm-started from a strong pretrained backbone (e.g., ImageNet-1K) and then adapted to local domains. Motivated by recent evidence that ReLU-like gating regimes (structural knowledge) stabilize earlier than the remaining parameter values (quantitative knowledge), we propose FedSQ (Federated Structural-Quantitative learning), a transfer-initialized neural federated procedure based on a DualCopy, piecewise-linear view of deep networks. FedSQ freezes a structural copy of the pretrained model to induce fixed binary gating masks during federated fine-tuning, while only a quantitative copy is optimized locally and aggregated across rounds. Fixing the gating reduces learning to within-regime affine refinements, which stabilizes aggregation under heterogeneous partitions. Experiments on two convolutional neural network backbones under i.i.d.\ and Dirichlet splits show that FedSQ improves robustness and can reduce rounds-to-best validation performance relative to standard baselines while preserving accuracy in the transfer setting.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。