通过时间一致性机制增强记忆,让多智能体强化学习更高效地探索和协作。
Episodic Memory Temporal Consistency for Cooperative Multi-Agent Reinforcement Learning

- 用时序一致嵌入和动态门控,防止记忆坍塌和错误奖励误导。
- 在SMAC和GRF上提升24%胜率,平均提升28%。
- 适合需要稳定协作的复杂多智能体场景,如游戏或机器人编队。
合作式多智能体强化学习常面临奖励稀疏和探索瓶颈。虽然情景记忆机制可通过重用高回报轨迹缓解此问题,但往往因激励分配无约束和语义表示坍塌导致智能体陷入局部最优。为此,我们提出情景记忆时间一致性(EMTC)框架,稳健构建并选择性利用历史经验。EMTC引入两个协同组件:(1) 时间一致性语义嵌入器,结合对比学习与时间条件状态重建,防止表示坍塌并实现精准记忆检索;(2) 时间一致性门控机制,基于时间一致性误差动态调节情景激励。该自适应门控可过滤伪成功轨迹带来的误导信号,有效缓解Q值过估计。我们提供理论保证,建立可观测时间一致性误差与底层轨迹最优性及表示质量间的严格误差界。在SMAC和GRF基准上的大量实验表明,EMTC持续优于现有最先进基线。特别地,相比最强的情景记忆基线,EMTC在超难SMAC场景中胜率提升高达24%,在GRF任务上平均提升28%。
原文摘要 · Abstract (English)
Cooperative Multi-Agent Reinforcement Learning (MARL) frequently suffers from severe reward sparsity and exploration bottlenecks. While episodic memory mechanisms mitigate these issues by reusing high-return trajectories, they often trap agents in local optima due to unconstrained incentive distribution and semantic representation collapse. To address this, we propose Episodic Memory Temporal Consistency (EMTC), a framework that robustly constructs and selectively leverages historical experiences. EMTC introduces two synergistic components: (1) a Temporally Consistent Semantic Embedder that integrates contrastive learning with time-conditioned state reconstruction, preventing representation collapse and enabling precise memory retrieval; and (2) a Temporal Consistency Gating Mechanism that dynamically modulates episodic incentives based on temporal consistency error. This adaptive gate filters misleading signals from pseudo-successful trajectories, effectively mitigating Q-value overestimation. We provide theoretical guarantees, establishing a strict error bound that directly links the observable temporal consistency error to the underlying trajectory optimality and representation quality. Extensive evaluations on the SMAC and GRF benchmarks demonstrate that EMTC consistently outperforms state-of-the-art baselines. Notably, compared to the strongest episodic baseline, EMTC achieves absolute win-rate improvements of up to 24% in super-hard SMAC scenarios and an average improvement of 28% across GRF tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。