用极少通信量实现高效联邦蒸馏，适合大规模低带宽场景。
TallyTrain: Communication-Efficient Federated Distillation
- 仅传输每个样本的最高概率类别索引，将通信量压缩至⌈log₂C⌉比特
- 在非独立同分布数据下表现优于软标签蒸馏，噪声过滤更有效
- 通信量减少1000倍以上，且性能超越主流联邦学习方法
联邦学习在两个正交维度上受带宽限制：模型规模限制了参数平均的频率，类别数量使得逐样本软标签蒸馏在大词表下不可行。随着系统规模扩大，这两项限制愈发严峻。本文提出TallyTrain，通过仅传输每个样本的argmax类别索引，将每轮通信的类别信息压缩至⌈log₂C⌉比特，其中C为输出类别数。该协议不仅显著压缩通信量，且在非独立同分布（non-IID）训练下优于软标签蒸馏，因为欠训练的客户端往往高度自信地犯错，而多数投票能有效过滤此类噪声，相比之下软标签平均会放大噪声。在多个标准基准测试中，TallyTrain以高达三个数量级更低的通信开销，达到或超越软标签蒸馏的性能。此外，我们还结合廉价硬标签共识与稀疏参数更新，提出带宽桥梁变体，其在所有测试操作点上均帕累托优于FedAvg、FedProx和FedDF基线。
原文摘要 · Abstract (English)
Federated learning is bandwidth-bound on two orthogonal axes: model size, which limits how often parameter-averaging methods can afford to merge, and class count, which makes per-probe soft-label distillation prohibitive at large vocabularies. Both ceilings tighten as modern systems scale. We collapse the class-count axis to $\lceil \log_2 C \rceil$ bits per probe by transmitting only each peer's $\arg\max$ class index, where $C$ is the number of output classes. The resulting protocol, TallyTrain, is not merely compressed: under non-IID training it can be preferable to soft-label distillation, because under-trained peers are confidently wrong and majority voting filters this noise where soft-label averaging amplifies it. Across standard benchmarks, TallyTrain matches or beats soft-label distillation at up to three orders of magnitude less communication. We also relax the model-size axis: we compose the cheap hard-label consensus with sparse parameter merges to obtain a bandwidth-bridge variant, which Pareto-dominates every tested operating point of the standard FedAvg, FedProx and FedDF baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。