通过知识蒸馏与原型学习，降低去中心化联邦学习通信开销。
ProFe: Communication-Efficient Decentralized Federated Learning via Distillation and Prototypes
- 用大模型知识训练小模型参与聚合，减少通信量。
- 引入原型机制提升对未知类别的识别能力，通信量降低40%-50%。
- 适合资源受限场景，兼顾通信效率与模型性能。
去中心化联邦学习（DFL）在保护隐私的同时避免了模型集中化风险，并缓解通信瓶颈。然而，在异构数据分布下，其仍面临通信管理与模型聚合效率的挑战。本文提出ProFe算法，融合知识蒸馏、原型学习与量化技术。利用本地大模型的知识训练小型模型用于聚合，引入原型以更好学习未见类别，并通过量化压缩通信数据。在MNIST、CIFAR10和CIFAR100等基准数据集上的实验表明，该方法可将通信成本降低约40%-50%，同时保持或提升模型性能。但因计算复杂度增加，训练时间额外延长约20%，存在效率与开销的权衡。
原文摘要 · Abstract (English)
Decentralized Federated Learning (DFL) trains models in a collaborative and privacy-preserving manner while removing model centralization risks and improving communication bottlenecks. However, DFL faces challenges in efficient communication management and model aggregation within decentralized environments, especially with heterogeneous data distributions. Thus, this paper introduces ProFe, a novel communication optimization algorithm for DFL that combines knowledge distillation, prototype learning, and quantization techniques. ProFe utilizes knowledge from large local models to train smaller ones for aggregation, incorporates prototypes to better learn unseen classes, and applies quantization to reduce data transmitted during communication rounds. The performance of ProFe has been validated and compared to the literature by using benchmark datasets like MNIST, CIFAR10, and CIFAR100. Results showed that the proposed algorithm reduces communication costs by up to ~40-50% while maintaining or improving model performance. In addition, it adds ~20% training time due to increased complexity, generating a trade-off.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。