解决多智能体在重尾通信与奖励下的协同优化问题,首次实现高鲁棒性决策。
Multi-agent Multi-armed Bandit with Fully Heavy-tailed Dynamics
- 利用重尾图中的枢纽结构设计噪声抑制机制,提升估计精度
- 同质场景下达到近似 $O(M^{1-rac{1}{α}} "log{T})$ 的后悔上界
- 适用于真实分布式系统中通信不稳、奖励异常的场景
研究完全重尾条件下的去中心化多智能体多臂老虎机问题,客户端通过稀疏随机图通信,其度分布具有重尾特性,且观测到的奖励分布为重尾(同质或异质),可能具有无限方差。目标是通过拉取全局最优动作来最大化系统性能。这是首个处理此类全重尾场景的工作,能刻画现实系统中多客户端间通信与推理的动态挑战。在同质情况下,算法框架利用重尾图中独特的枢纽结构,通过枢纽估计器聚合奖励以降低噪声,构造UCB指数;在M个客户端、度分布幂律指数α>1时,实现几乎为$O(M^{1-rac{1}{α}} "log{T})$的后悔上界。在异质奖励下,客户端通过邻居通信同步,聚合交换的估计量用于UCB指数;结合在稀疏随机图上新建立的信息延迟界,证明了$O(M "log{T})$的后悔上界。结果优于现有工作,后者仅考虑时不变连通图或密集图中的轻尾动态与奖励。
原文摘要 · Abstract (English)
We study decentralized multi-agent multi-armed bandits in fully heavy-tailed settings, where clients communicate over sparse random graphs with heavy-tailed degree distributions and observe heavy-tailed (homogeneous or heterogeneous) reward distributions with potentially infinite variance. The objective is to maximize system performance by pulling the globally optimal arm with the highest global reward mean across all clients. We are the first to address such fully heavy-tailed scenarios, which capture the dynamics and challenges in communication and inference among multiple clients in real-world systems. In homogeneous settings, our algorithmic framework exploits hub-like structures unique to heavy-tailed graphs, allowing clients to aggregate rewards and reduce noises via hub estimators when constructing UCB indices; under $M$ clients and degree distributions with power-law index $α> 1$, our algorithm attains a regret bound (almost) of order $O(M^{1 -\frac{1}α} \log{T})$. Under heterogeneous rewards, clients synchronize by communicating with neighbors, aggregating exchanged estimators in UCB indices; With our newly established information delay bounds on sparse random graphs, we prove a regret bound of $O(M \log{T})$. Our results improve upon existing work, which only address time-invariant connected graphs, or light-tailed dynamics in dense graphs and rewards.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。