动态分配邻居权重，加速去中心化学习且开销极小。
Efficiency Boost in Decentralized Optimization: Reimagining Neighborhood Aggregation with Minimal Overhead
- 根据邻居损失动态分配权重，优先选择信息差异大的节点。
- 在MNIST/CIFAR10/CIFAR100上训练速度显著提升，适配多种网络拓扑。
- 兼容任意优化算法，无需额外通信与内存开销，易集成。
在数据敏感的分布式学习场景中，去中心化架构因强化隐私保护与提升计算效率而愈发重要。本文提出DYNAWEIGHT框架，用于多智能体网络中的信息聚合。该方法通过动态调整邻居服务器的权重，依据其本地数据集上的相对损失值，优先选择信息差异较大的节点，尤其在数据异构性强的场景下表现突出。实验在MNIST、CIFAR10和CIFAR100上进行，覆盖不同服务器数量与图结构，均验证了训练速度的显著提升。DYNAWEIGHT作为通用聚合方案，可与任何底层优化算法协同工作，具备低通信与内存开销，展现出高度的灵活性与部署潜力。
原文摘要 · Abstract (English)
In today's data-sensitive landscape, distributed learning emerges as a vital tool, not only fortifying privacy measures but also streamlining computational operations. This becomes especially crucial within fully decentralized infrastructures where local processing is imperative due to the absence of centralized aggregation. Here, we introduce DYNAWEIGHT, a novel framework to information aggregation in multi-agent networks. DYNAWEIGHT offers substantial acceleration in decentralized learning with minimal additional communication and memory overhead. Unlike traditional static weight assignments, such as Metropolis weights, DYNAWEIGHT dynamically allocates weights to neighboring servers based on their relative losses on local datasets. Consequently, it favors servers possessing diverse information, particularly in scenarios of substantial data heterogeneity. Our experiments on various datasets MNIST, CIFAR10, and CIFAR100 incorporating various server counts and graph topologies, demonstrate notable enhancements in training speeds. Notably, DYNAWEIGHT functions as an aggregation scheme compatible with any underlying server-level optimization algorithm, underscoring its versatility and potential for widespread integration.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。