让多智能体学会识别任务类型,实现一练多用的通用策略。
Trajectory-Class-Aware Multi-Agent Reinforcement Learning
- 通过轨迹嵌入和聚类识别任务类型,增强策略感知能力。
- 在星际争霸Ⅱ多任务场景中,性能超越现有最佳方法。
- 适合需要通用协作能力的多智能体系统研究者。
在多智能体强化学习中,泛化能力是解决多种任务的关键挑战,这些任务可能需要不同的联合策略或协调机制,而无需为每项任务专门训练策略。我们称此类问题为多任务问题,并通过单一训练过程使智能体具备在多任务环境中通用的能力。为此,提出轨迹类别感知的多智能体强化学习(TRAMA)。在TRAMA中,智能体通过部分观测识别其经历的轨迹类别,利用该轨迹感知信息作为动作策略的额外输入。具体包含三个核心目标:(a) 构建量化隐空间以生成反映轨迹间关键相似性的轨迹嵌入;(b) 使用这些嵌入进行轨迹聚类;(c) 构建轨迹类别感知策略。针对第(c)点,引入代理级轨迹类别预测器,并为每类轨迹设计轨迹类别表示模型。每个智能体基于该类别表示与自身部分观测共同决定行动,实现任务感知执行。所提方法在多个任务上评估,包括基于星际争霸Ⅱ构建的多任务场景,实验结果表明其性能进一步优于当前最先进基线。
原文摘要 · Abstract (English)
In the context of multi-agent reinforcement learning, generalization is a challenge to solve various tasks that may require different joint policies or coordination without relying on policies specialized for each task. We refer to this type of problem as a multi-task, and we train agents to be versatile in this multi-task setting through a single training process. To address this challenge, we introduce TRajectory-class-Aware Multi-Agent reinforcement learning (TRAMA). In TRAMA, agents recognize a task type by identifying the class of trajectories they are experiencing through partial observations, and the agents use this trajectory awareness or prediction as additional information for action policy. To this end, we introduce three primary objectives in TRAMA: (a) constructing a quantized latent space to generate trajectory embeddings that reflect key similarities among them; (b) conducting trajectory clustering using these trajectory embeddings; and (c) building a trajectory-class-aware policy. Specifically for (c), we introduce a trajectory-class predictor that performs agent-wise predictions on the trajectory class; and we design a trajectory-class representation model for each trajectory class. Each agent takes actions based on this trajectory-class representation along with its partial observation for task-aware execution. The proposed method is evaluated on various tasks, including multi-task problems built upon StarCraft II. Empirical results show further performance improvements over state-of-the-art baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。