无通信下多智能体如何高效找出最优的K个组合策略
The Price of Decentralization in Top-$K$ Arm Identification
- 设计无需通信的消除算法,利用不同观测条件下的残余信号实现隐式协作
- 在完全不对称观测下样本复杂度增加4倍,且该增益不可避免
- 适用于分布式决策、资源分配等需协同选优但无法通信的场景
在多智能体多臂赌博机中,团队需在每轮由各智能体独立选择动作组成联合动作,并最终选出均值奖励最高的前K个联合动作。难点在于每个智能体仅能观测自身动作和奖励,无法看到他人行为或收益。本文研究三种观测情形:(A) 共享奖励但隐藏动作,(B) 可观测动作但奖励私有,(C) 完全不对称。提出无需通信的消除算法(UCB-Intervals),分别利用共享动作排序、可观测偏差和扩大置信半径来重建协调机制。在固定预算与固定置信度目标下给出匹配分析,将三类情形统一为依赖乘数c与共识因子ρ的元保证。核心结论为:共享奖励情形近似最优,而移除通信带来的统计代价是样本复杂度乘以ρ²,完全不对称下为固定4倍惩罚。停止时间量级为O(∑_a log(A^M/δ)/Δ_a²),固定预算误差为exp(−Θ(T/H₁)),其中联合动作数A^M的影响被证明不可避免。
原文摘要 · Abstract (English)
Cooperative teams often need to agree on the best few options rather than simply accumulate reward, and they must do so while each member sees only a fragment of the team's collective experience. We study this as top-$K$ joint-arm identification in multi-agent multi-armed bandits: at every round $M$ agents simultaneously choose individual actions that compose a joint arm, and the team must ultimately return the $K$ joint arms of highest mean reward. The difficulty is that an agent may not observe the actions of others, their rewards, or either. We treat three observability regimes---(A) shared rewards with hidden actions, (B) observed actions with private rewards, and (C) full asymmetry---and design communication-free elimination algorithms (UCB-Intervals) that reconstruct implicit coordination from whatever signal each regime leaves intact: a shared arm ordering in (A), observable deviations in (B), and enlarged confidence radii under (C). We give matching analyses in both the fixed-budget and fixed-confidence objectives, then fold all three regimes into a single meta-guarantee indexed by a multiplicity $c$ and a consensus factor $ρ$. Our central result is quantitative rather than merely algorithmic: change-of-measure lower bounds show that shared-reward identification is optimal up to one universal logarithmic factor, and that the entire statistical price of removing communication is a multiplicative $ρ^2$ in sample complexity---a fixed $4\times$ penalty under full asymmetry. The resulting stopping time scales as $O\!\left(\sum_{\mathbf{a}} \frac{\log(A^M/δ)}{Δ_{\mathbf{a}}^2}\right)$ and the fixed-budget error as $\exp(-Θ(T/H_1))$, with the dependence on the joint-action count $A^M$ shown to be unavoidable.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。