针对异构数据与节点，提出高效个性化分布式学习新方法。
Communication-Efficient Personalized Distributed Learning with Data and Node Heterogeneity
- 用二值掩码与实数参数的哈达玛积表示本地模型，实现轻量更新。
- 通过结构化稀疏正则化降低硬件复杂度，提升通信效率。
- 适合资源受限的异构设备，尤其适用于个性化需求强的场景。
为解决去中心化学习中数据与节点异构带来的挑战，本文提出分布式强彩票假设（DSLTH），并基于此设计了一种通信高效的个性化学习算法。每个本地模型由全局实数参数与个性化二值掩码的哈达玛积表示，本地模型通过更新和融合个性化二值掩码实现学习，而实数参数在不同代理间保持固定。为进一步降低硬件实现复杂度，损失函数中引入组稀疏正则项，使学习后的本地模型具备结构化稀疏性。通过引入中间聚合张量并加入个性化微调步骤，设计了二值掩码聚合算法，约束模型更新以贴近本地数据分布。所提方法有效利用了代理间的关联性，在异构节点条件下满足个性化需求。同时提供了DSLTH的理论证明，奠定了方法基础。数值实验验证了DSLTH的有效性，并展示了算法性能。
原文摘要 · Abstract (English)
To jointly tackle the challenges of data and node heterogeneity in decentralized learning, we propose a distributed strong lottery ticket hypothesis (DSLTH), based on which a communication-efficient personalized learning algorithm is developed. In the proposed method, each local model is represented as the Hadamard product of global real-valued parameters and a personalized binary mask for pruning. The local model is learned by updating and fusing the personalized binary masks while the real-valued parameters are fixed among different agents. To further reduce the complexity of hardware implementation, we incorporate a group sparse regularization term in the loss function, enabling the learned local model to achieve structured sparsity. Then, a binary mask aggregation algorithm is designed by introducing an intermediate aggregation tensor and adding a personalized fine-tuning step in each iteration, which constrains model updates towards the local data distribution. The proposed method effectively leverages the relativity among agents while meeting personalized requirements in heterogeneous node conditions. We also provide a theoretical proof for the DSLTH, establishing it as the foundation of the proposed method. Numerical simulations confirm the validity of the DSLTH and demonstrate the effectiveness of the proposed algorithm.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。