多智能体强化学习中,高效识别各组最优动作并减少通信开销。
Near Optimal Best Arm Identification for Clustered Bandits
- 先聚类后找最优动作,或先找最优再聚类,分阶段优化决策。
- 在小集群数时,算法样本复杂度接近理论最优,显著降低资源消耗。
- 适用于大规模多智能体系统,尤其适合集群远少于个体的场景。
本文研究多智能体多臂赌博机中的最优动作识别问题。考虑将 $N$ 个智能体划分为 $M$ 个簇,每个簇独立解决一个随机赌博机问题,且智能体与动作的映射关系未知。每个动作对应 $K$ 个可选臂,目标是在 $δ$-概率正确($δ$-PC)框架下,以最小化样本复杂度和通信开销的方式,为每个智能体识别出最优臂。本文提出两种新算法:先聚类后最优动作识别(Cl-BAI)与先最优动作识别后聚类(BAI-Cl)。Cl-BAI 先基于学习任务对智能体聚类,再分别识别各簇最优臂;BAI-Cl 则反序进行。两者均采用逐次消除框架,保证计算效率与高精度。本文建立了两者的 $δ$-PC 保证,推导了样本复杂度上界,并给出了该问题类的下界。当 $M$ 为常数时,BAI-Cl 的变体在阶数意义下达到极小最大样本复杂度。在合成数据及真实数据集(MovieLens、Yelp)上的实验表明,所提算法在样本与通信效率方面表现优异,尤其在 $M \ ext{\ll} N$ 场景下优势明显。
原文摘要 · Abstract (English)
This work investigates the problem of best arm identification for multi-agent multi-armed bandits. We consider $N$ agents grouped into $M$ clusters, where each cluster solves a stochastic bandit problem. The mapping between agents and bandits is a priori unknown. Each bandit is associated with $K$ arms, and the goal is to identify the best arm for each agent under a $δ$-probably correct ($δ$-PC) framework, while minimizing sample complexity and communication overhead. We propose two novel algorithms: Clustering then Best Arm Identification (Cl-BAI) and Best Arm Identification then Clustering (BAI-Cl). Cl-BAI uses a two-phase approach that first clusters agents based on the bandit problems they are learning, followed by identifying the best arm for each cluster. BAI-Cl reverses the sequence by identifying the best arms first and then clustering agents accordingly. Both algorithms leverage the successive elimination framework to ensure computational efficiency and high accuracy. We establish $δ$-PC guarantees for both methods, derive bounds on their sample complexity, and provide a lower bound for this problem class. Moreover, when $M$ is small (a constant), we show that the sample complexity of a variant of BAI-Cl is minimax optimal in an order-wise sense. Experiments on synthetic and real-world datasets (MovieLens, Yelp) demonstrate the superior performance of the proposed algorithms in terms of sample and communication efficiency, particularly in settings where $M \ll N$.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。