用信念分布差异决定通信时机,让多智能体沟通更智能稳定。
When to Communicate: Belief Distributions and KL Divergence for Principled Gating in Multi-Agent RL

- 基于信念分布的KL散度动态判断通信时机,避免盲目发送。
- 在20×20猎物追捕任务中,阈值0.5时成功率42%,优于基线11个百分点。
- 即使不通信,信念头也提升协作表现,适合需要高效协同的场景。
多智能体强化学习中,有效通信不仅需决定传什么,还需决定何时传。现有方法或每轮通信,或通过高方差的REINFORCE策略学习二元门控,导致行为不稳定且不可解释。本文提出一种原则性替代方案:仅当智能体间信念分布的KL散度超过固定阈值时才通信。每个智能体维护一个基于LSTM隐状态的软最大信念分布,仅在信念分歧足够大时交换信息。在IC3Net的猎物追捕基准(两种环境规模,各5个种子)和MPE simple_spread上评估,对比IC3Net、CommNet及独立控制器。在10×10环境中IC3Net略优;在更难的20×20环境中,阈值ε∈{0.1,0.3,0.5,1.0}的消融实验显示倒U型曲线:ε=0.5时平均步数73.84,成功率42%,相较IC3Net的75.31步与31%成功率,分别领先1.47步与11个百分点,且种子方差更小。在MPE上,信念头使平均奖励提升12分,方差降低26倍,即便门控关闭亦然,表明其具备双重贡献:信念收敛时实现合理门控,且改善隐表示促进协作。
原文摘要 · Abstract (English)
Effective communication in multi-agent reinforcement learning requires agents to decide not only \textit{what} to communicate, but when? Existing approaches either communicate at every timestep or learn a binary gate through REINFORCE policy gradients \cite{singh2019}, a high-variance signal that produces unstable and uninterpretable gating behavior. I propose a principled alternative: agents communicate only when the KL divergence between their learned belief distributions exceeds a fixed threshold. Each agent maintains a belief distribution over a latent world state computed as a softmax over its LSTM hidden state, and communicates only when belief disagreement is large enough to justify information exchange. I evaluate this approach on the Predator-Prey benchmark from IC3Net \cite{singh2019} across two environment sizes with 5 seeds each, and on MPE simple\_spread \cite{lowe2017}, comparing against IC3Net, CommNet, and an independent controller. On PP 10$\times$10, IC3Net outperforms KL-belief at all thresholds. On the harder PP 20$\times$20, a threshold ablation over $\varepsilon \in \{0.1, 0.3, 0.5, 1.0\}$ reveals an inverted U-shape: $\varepsilon=0.5$ achieves 73.84 average steps and 42\% success rate versus IC3Net's 75.31 steps and 31\%, a gap of 1.47 steps and 11 percentage points with tighter seed variance. On MPE, the belief head improves mean reward by 12 points and reduces variance by 26$\times$ even when gating is inactive, suggesting two orthogonal contributions: principled gating when beliefs can converge, and improved latent representations that benefit coordination regardless.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。