解决多智能体强化学习中状态空间不匹配的迁移难题。
ASALT: Adaptive State Alignment for Lateral Transfer in Multi-agent Reinforcement Learning

- 引入观察与状态双适配器,映射异构状态空间到共享嵌入空间。
- 在合作场景下提升样本效率和全局回报,优于现有基线方法。
- 有效缓解因观测与动作空间差异导致的负迁移问题。
多智能体强化学习(MARL)致力于训练多个追求协作、竞争或混合目标的智能体。以往研究探讨了源域与目标域间的迁移学习,但多数方法要求观察空间与全局状态空间维度一致。本文提出ASALT方法,显式处理源域与目标域间状态空间维度不匹配的问题。该方法通过观察级与状态级适配器,将目标域的观测与全局状态映射至共享嵌入空间,从而实现对策略网络与价值网络的有效知识迁移。适配器生成的嵌入支持在异构域间高效转移策略。在标准基准环境的多种配置中,实验表明ASALT在合作设置下显著提升样本效率与全局回报,性能优于现有基线;其效果取决于源域与目标域间的维度差异程度。此外,研究发现ASALT能有效缓解负迁移,而负迁移常是跨不同观测与动作空间迁移策略时的主要障碍。
原文摘要 · Abstract (English)
Multi-agent reinforcement learning (MARL) addresses the problem of training multiple agents that pursue collaborative, competitive, or mixed objectives. Prior work has investigated transfer learning between source and target domains in MARL; however, the majority of existing approaches impose the constraint that the dimensionalities of the observation space and the global state space must be identical across domains. In this paper, we introduce a method that explicitly accommodates mismatched state-space dimensionalities between source and target domains. The proposed approach, ASALT, incorporates both observation-level and state-level adapters that map the target-domain observations and global states into a shared embedding space, thereby enabling more effective transfer of knowledge across both actors and critics. These adapters can generate embeddings that support efficient strategy transfer across heterogeneous domains. Experimental results on multiple configurations in standard benchmark environments demonstrate that ASALT surpasses existing baselines in terms of sample efficiency and global return in cooperative settings, but its effectiveness depends on the degree of mismatch between source and target domains. Furthermore, our findings indicate that ASALT mitigates negative transfer, which frequently constitutes a major obstacle when transferring policies between domains with differing observation and action spaces.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。