动态分配探索预算,让靠谱的智能体多探索。
Quality-Aware Exploration Budget Allocation for Cooperative Multi-Agent Reinforcement Learning

- 用信号质量自动分配探索强度,避免无效尝试。
- 在7个基准任务中均达到顶尖性能,最高提升18%。
- 适合需要协作的多智能体系统,尤其对噪声敏感场景。
协作式多智能体强化学习(MARL)需在组合爆炸的状态-动作空间中发现联合策略,但有效协调配置极为稀少。内在动机通过在任务奖励中加入新颖性奖励来驱动探索,其效果依赖探索强度参数β:过大则淹没任务信号导致协调崩溃,过小则无法发现罕见策略。本文解决两个互补挑战:全局调整β随训练进程变化,以及根据各智能体内在奖励信号的可靠性分配探索预算。框架结合返回条件的S型调度(RCB)实现全局强度控制,引入每智能体奖励信号质量(RSQ)指标,将探索预算集中在信号可靠的智能体上。核心思想是:接收噪声内在奖励的智能体应减少探索强度,该分配可由信噪比统计自动确定。后续距离(SD)作为准度量内在奖励,天然生成可区分的每智能体信号质量,完整框架具备收敛性和顺序保持性保证。在七个协作基准(MPE、SMAX、MABrax)上,本方法在所有环境中均取得顶级回报。
原文摘要 · Abstract (English)
Cooperative multi-agent reinforcement learning (MARL) requires agents to discover joint strategies in a combinatorially large state-action space, yet effective coordination configurations are exceedingly rare. Intrinsic motivation, which augments task rewards with novelty bonuses, is a popular approach for driving exploration, but its effectiveness hinges on the exploration intensity $β$, where too large a value overwhelms the task signal and causes coordination collapse, while too small a value prevents discovery of rare strategies. We address two complementary challenges: adapting $β$ globally over training, and allocating the exploration budget across agents whose intrinsic reward signals vary in reliability. Our framework combines a return-conditioned sigmoid schedule (RCB) for global intensity control with a per-agent Reward Signal Quality (RSQ) metric that concentrates the exploration budget on agents with reliable signals. The core insight is that agents receiving noisy intrinsic rewards should explore less aggressively, and this allocation can be determined automatically from signal-to-noise statistics. Successor Distance (SD), a quasimetric intrinsic reward, naturally produces distinguishable per-agent signal quality, completing the framework with convergence and ordering preservation guarantees. On seven cooperative benchmarks (MPE, SMAX, MABrax), our method achieves top-tier returns across all environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。