让智能体预测通信中断时的状态,只关注高回报的关键信息。
Value-Aware Prediction for Robust Multi-Agent Coordination Under Communication Loss

- 用策略价值动态加权预测损失,聚焦高回报状态变化。
- 通信可靠性低于40%时,平均收益提升超20%,方差降低64.7%。
- 适合通信不稳定场景下的多智能体协同任务,如机器人集群。
真实场景中多智能体协同严重依赖通信,但常因物理与环境限制而中断。为应对间歇性通信故障,智能体可使用内部预测模型估算缺失的共享状态。然而,传统基于重建目标训练的预测器对所有状态转移同等对待,在强化学习中会浪费容量学习随机探索噪声和次优策略的过时动态。本文提出价值感知的多智能体观测共享框架(Value-Aware MARO),通过利用底层演员-评论家架构生成的优势估计值动态加权预测器的损失函数,使预测学习过程与策略演化显式耦合。该方法将模型容量集中于智能体主动强化的高回报动态。我们在多智能体粒子环境中评估了不同通信可靠性水平下的性能。实验表明,当通信可靠性下降至40%以下时,本方法仍能保持性能;在基础模型已表现良好的任务中,其有效防止了标准预测器在高丢包率下的性能崩溃。在这些场景下,平均回报提升超过20%,性能方差平均降低64.7%。
原文摘要 · Abstract (English)
Robust multi-agent coordination relies heavily on inter-agent communication, which is frequently disrupted by physical and environmental constraints in real-world deployments. To maintain operation during these intermittent communication failures, agents can employ internal prediction models to estimate missing shared state information. However, predictors trained with standard reconstruction objectives treat all transitions equally. In a Reinforcement Learning context, this forces the model to waste capacity learning stochastic exploration noise and the outdated dynamics of suboptimal policies. In this paper, we propose a value-aware extension of Multi-Agent Observation Sharing under Communication Dropout (MARO) to patch communication gaps; we refer to this method as Value-Aware MARO. By dynamically weighting the predictor's loss function using advantage estimates derived from the underlying actor-critic architecture, our objective explicitly couples the predictor's learning process to the policy's evolution. This formulation focuses the model's capacity on the intentional, high-return dynamics actively reinforced by the agents. We evaluate our framework on several tasks within the Multi-Agent Particle Environment under varying communication reliability levels. Experimental results demonstrate that our approach maintains performance under declining communication reliability, particularly below 40%. While our method performs comparably in tasks where the baseline already maintains high coordination, our value-aware weighting effectively prevents the performance collapse observed in the standard predictor during high-attrition scenarios. In these environments, our method achieves an average improvement in mean returns of more than 20% and reduces performance variance by a mean of 64.7% compared to the standard unweighted baseline.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。