多智能体强化学习中,通信随机化下实现近最优累积损失控制。
Distributed Multi-Agent Bandits Over Erdős-Rényi Random Networks
- 基于随机共识与淘汰机制设计分布式算法
- 理论证明后悔上界为 log T 阶,包含通信效率影响项
- 适合研究分布式决策与网络鲁棒性的研究人员
研究在异构奖励下,通过随机动态通信图的分布式多智能体多臂赌博机问题。每个时间步,智能体在由固定连通基图 $G$ 生成的埃拉托斯蒂尼-雷尼随机图 $G_t$ 上通信,每条边以概率 $p$ 独立存在。各智能体同一动作的奖励分布不同,目标是最小化相对于全局均值奖励的累积期望后悔。提出一种完全分布式算法,结合动作淘汰策略与随机对偶共识。理论上证明后悔上界为 $O(\log T)$,包含集中式最优后悔 $O\left(\sum_{k: Δ_k>0} \frac{\log T}{Δ_k}\right)$ 和额外项 $O\left(\frac{N^2 \log T}{p λ_{N-1}(Lap(G))} + \frac{KN^2 \log T}{p}\right)$,其中 $N$、$K$ 分别为智能体数和动作数,$λ_{N-1}(Lap(G))$ 为图 $G$ 的代数连通度。该结果揭示了通信效率与后悔之间的根本权衡。进一步给出近乎最优的后悔下界。数值实验验证算法优于现有基准,并证实理论后悔增长规律。
原文摘要 · Abstract (English)
We study the distributed multi-agent multi-armed bandit problem with heterogeneous rewards over random communication graphs. Uniquely, at each time step $t$ agents communicate over a time-varying random graph $G_t$ generated by applying the Erdős-Rényi model to a fixed connected base graph $G$ (for classical Erdős-Rényi graphs, $G$ is a complete graph), where each potential edge in $G$ is randomly and independently present with the link probability $p$. Notably, the resulting random graph is not necessarily connected at each time step. Each agent's arm rewards follow time-invariant distributions, and the reward distribution for the same arm may differ across agents. The goal is to minimize the cumulative expected regret relative to the global mean reward of each arm, defined as the average of that arm's mean rewards across all agents. To this end, we propose a fully distributed algorithm that integrates the arm elimination strategy with the random gossip algorithm. We theoretically show that the regret upper bound is of order $\log T$ and is highly interpretable, where $T$ is the time horizon. It includes the optimal centralized regret $O\left(\sum_{k: Δ_k>0} \frac{\log T}{Δ_k}\right)$ and an additional term $O\left(\frac{N^2 \log T}{p λ_{N-1}(Lap(G))} + \frac{KN^2 \log T}{p}\right)$ where $N$ and $K$ denote the total number of agents and arms, respectively. This term reflects the impact of $G$'s algebraic connectivity $λ_{N-1}(Lap(G))$ and the link probability $p$, and thus highlights a fundamental trade-off between communication efficiency and regret. As a by-product, we show a nearly optimal regret lower bound. Finally, our numerical experiments not only show the superiority of our algorithm over existing benchmarks, but also validate the theoretical regret scaling with problem complexity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。