用高斯混合与Transformer提升元强化学习在动态多任务环境下的适应性
TIMRL: A Novel Meta-Reinforcement Learning Framework for Non-Stationary and Multi-Task Environments
- 用高斯混合模型和Transformer构建任务推理机制，实现显式任务编码
- 在非平稳多任务环境中显著提升采样效率，任务分类准确率高
- 适合需要快速适应新任务的机器人控制等动态场景
近年来，元强化学习（meta-RL）算法被提出以提高决策与控制领域的样本效率，使智能体能够从少量样本中学习新知识。然而，大多数研究使用高斯分布提取任务表示，难以适应非平稳环境中的任务变化。为此，我们提出一种新型元强化学习方法，结合高斯混合模型与Transformer网络构建任务推理模型。高斯混合模型扩展了任务表示并实现任务的显式编码；具体而言，通过Transformer网络对任务分类进行编码，以确定对应的任务高斯分量。利用任务标签，Transformer网络采用监督学习进行训练。我们在包含非平稳性和多任务特性的MuJoCo基准测试上验证该方法。实验结果表明，所提方法显著提升了样本效率，并能准确识别任务分类，在环境中表现优异。
原文摘要 · Abstract (English)
In recent years, meta-reinforcement learning (meta-RL) algorithm has been proposed to improve sample efficiency in the field of decision-making and control, enabling agents to learn new knowledge from a small number of samples. However, most research uses the Gaussian distribution to extract task representation, which is poorly adapted to tasks that change in non-stationary environment. To address this problem, we propose a novel meta-reinforcement learning method by leveraging Gaussian mixture model and the transformer network to construct task inference model. The Gaussian mixture model is utilized to extend the task representation and conduct explicit encoding of tasks. Specifically, the classification of tasks is encoded through transformer network to determine the Gaussian component corresponding to the task. By leveraging task labels, the transformer network is trained using supervised learning. We validate our method on MuJoCo benchmarks with non-stationary and multi-task environments. Experimental results demonstrate that the proposed method dramatically improves sample efficiency and accurately recognizes the classification of the tasks, while performing excellently in the environment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。