提出更优的分布式专家学习协议，显著降低通信开销。
Better Bounds for the Distributed Experts Problem
- 设计新算法，在保持低损失的前提下减少通信量。
- 在 T 步内实现约 1/√T 的遗憾边界，优于先前方法。
- 适合大规模分布式学习场景，尤其关注通信效率的用户。
本文研究分布式专家问题，其中 n 个专家分布在 s 个服务器上，持续 T 个时间步。每个专家在每一步 t 的损失为各服务器上损失向量的 ℓ_p 范数。目标是最小化遗憾 R，即分布式协议总损失与最优专家损失之差，且通信量最小。本文提出一种协议，遗憾约为 R ≳ 1/(√T·polylog(nsT))，通信量为 O((n/R² + s/R²)·max(s^{1-2/p},1)·polylog(nsT)) 位，优于以往工作。
原文摘要 · Abstract (English)
In this paper, we study the distributed experts problem, where $n$ experts are distributed across $s$ servers for $T$ timesteps. The loss of each expert at each time $t$ is the $\ell_p$ norm of the vector that consists of the losses of the expert at each of the $s$ servers at time $t$. The goal is to minimize the regret $R$, i.e., the loss of the distributed protocol compared to the loss of the best expert, amortized over the all $T$ times, while using the minimum amount of communication. We give a protocol that achieves regret roughly $R\gtrsim\frac{1}{\sqrt{T}\cdot\text{poly}\log(nsT)}$, using $\mathcal{O}\left(\frac{n}{R^2}+\frac{s}{R^2}\right)\cdot\max(s^{1-2/p},1)\cdot\text{poly}\log(nsT)$ bits of communication, which improves on previous work.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。