让智能体通过因果知识快速适应动态环境变化,无需重新训练。
Causal Knowledge Transfer for Multi-Agent Reinforcement Learning in Dynamic Environments
- 用因果干预建模碰撞恢复动作,生成可转移的宏观策略
- 异构目标智能体在新环境中达成约50%的性能提升
- 适合需要快速适应复杂动态环境的多智能体系统
多智能体强化学习(MARL)在需协同行为的环境中已取得显著进展,但在目标不断变化的非平稳环境中,知识迁移仍具挑战。传统方法难以泛化,智能体常需高昂代价重新训练。本文提出一种因果知识迁移框架,使智能体学习并共享环境路径的紧凑因果表示。当环境变化(如新增障碍物)导致碰撞时,将每次碰撞视为一个因果干预,对应一系列恢复动作(宏操作),其效果体现为绕过障碍物并提高达成目标(最大化累积奖励)的概率。该恢复动作宏可由另一智能体在线传输,并以零样本方式应用——仅需根据局部上下文(碰撞信息)查询查找模型即可。实验显示:(1) 具有异构目标的智能体在新环境中,性能可弥补随机探索与完全重训练策略之间约一半的差距;(2) 因果知识迁移的效果取决于环境复杂度与智能体目标异质性的相互作用。
原文摘要 · Abstract (English)
[Context] Multi-agent reinforcement learning (MARL) has achieved notable success in environments where agents must learn coordinated behaviors. However, transferring knowledge across agents remains challenging in non-stationary environments with changing goals. [Problem] Traditional knowledge transfer methods in MARL struggle to generalize, and agents often require costly retraining to adapt. [Approach] This paper introduces a causal knowledge transfer framework that enables RL agents to learn and share compact causal representations of paths within a non-stationary environment. As the environment changes (new obstacles), agents' collisions require adaptive recovery strategies. We model each collision as a causal intervention instantiated as a sequence of recovery actions (a macro) whose effect corresponds to a causal knowledge of how to circumvent the obstacle while increasing the chances of achieving the agent's goal (maximizing cumulative reward). This recovery action macro is transferred online from a second agent and is applied in a zero-shot fashion, i.e., without retraining, just by querying a lookup model with local context information (collisions). [Results] Our findings reveal two key insights: (1) agents with heterogeneous goals were able to bridge about half of the gap between random exploration and a fully retrained policy when adapting to new environments, and (2) the impact of causal knowledge transfer depends on the interplay between environment complexity and agents' heterogeneous goals.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。