让不同算力的设备共享一个弹性模型,高效适配各自计算预算。
FEAST: Federated Shared-Space Training for Resource-Heterogeneous Clients

- 在每台设备预算内联合训练多个子网络,共享参数空间。
- 在596M MACs下准确率达71.06%,比最强基线高2.4个百分点。
- 支持按设备算力动态分配模型大小,适合边缘设备部署。
联邦学习需服务计算能力各异的设备。固定模型难以适配所有设备,而为每次部署单独训练模型成本过高。联邦超网络训练通过学习一个可扩展模型,包含不同规模的子网络,再将合适子网络部署至各设备。然而,当客户端推理预算不同时,仅高成本子网络独有的参数能被少数设备访问。本文提出FEAST,一种联邦共享空间训练框架,在客户端预算内联合训练多个子网络;预算定制的子超网络路由仅传输相关参数部分,稀疏聚合合并返回的参数片段。训练后的超网络可直接服务于联邦中使用的子网络,并支持事后提取额外子网络而无需重新联邦训练。我们进一步发现,独立分配客户端数据量与推理预算会扭曲准确率与推理成本的对比关系,提出单参数γ分配协议以控制该耦合。实验显示,SuperFedNAS和DeepFedNAS在2500万和5.96亿次推理MACs下的准确率分别仅达随机水平和17.09%;而FEAST在5.96亿次下达到71.06%,较最大层级最强模型-异构权重共享基线高出2.4个百分点。在CIFAR-100、CINIC-10和TinyImageNet-200上,当每个客户端获得其可负担的最大子网络时,FEAST在所评估的权重共享方法中实现最高群体平均准确率。子超网络路由相比全超网络传输,将总参数流量降低6.8倍。
原文摘要 · Abstract (English)
Federated learning (FL) must serve devices with varying computational capabilities. A fixed model cannot suit all devices, while training one model per deployment limit is costly. Federated supernet training instead learns one elastic model with differently sized subnetworks, then deploys a suitable one to each device. When client inference budgets differ, however, parameters exclusive to high-cost subnetworks are reachable by fewer clients. We propose FEAST, a federated shared-space training framework that counters this imbalance by jointly training multiple subnetworks within each client's limit. Budget-tailored sub-supernet routing sends only the relevant supernet portion, and sparse aggregation merges the returned parameter slices. The trained supernet directly serves the subnetworks used during federation and supports post-hoc extraction of additional subnetworks without federated retraining. We further show that independently assigning clients' training-data volumes and inference budgets can distort accuracy--inference-cost comparisons in heterogeneous FL simulations, and introduce a one-parameter $γ$-allocation protocol to control this coupling. In our experimental setup, the SuperFedNAS and DeepFedNAS supernet training procedures remain near chance at 25M and reach at most $17.09\%$ at $596$M inference MACs; FEAST reaches $71.06\%$ at $596$M, $2.4$ points above the strongest model-heterogeneous weight-sharing baseline at its largest tier. Across CIFAR-100, CINIC-10, and TinyImageNet-200, FEAST achieves the highest population-averaged accuracy among the evaluated weight-sharing methods when each client receives its largest affordable subnetwork. Sub-supernet routing reduces aggregate model-parameter traffic by $6.8\times$ relative to full-supernet transmission.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。