用可解释的程序图实现连续多任务强化学习，自动切换行为应对复杂环境。
Multi-action Tangled Program Graphs for Multi-task Reinforcement Learning with Continuous Control
- 构建基于程序图的多动作框架，通过控制流调度多个子策略。
- 在五种随机障碍的半猫环境中表现优于基线，结合分层选择提升性能。
- 模型决策路径完全可读，适合需要透明性的工业级应用。
过去几十年中，机器学习被广泛用于学习复杂任务。强化学习（RL）受人类行为启发，通过为特定任务开发特定行为来实现目标。为进一步挑战算法，多任务强化学习（MTRL）环境被提出，要求单一模型学习多种行为。现有的纠缠程序图（TPG）算法是为离散型MTRL设计的遗传编程（GP）方法。最近，MAPLE算法作为另一种GP方法，在单任务连续强化学习中表现出色。本文提出一种改进版本——多动作纠缠程序图（MATPG），通过聚合多个MAPLE智能体并构建控制流以激活它们。最初仅在单任务环境中测试，其表现与MAPLE相当。本文构建了一个基于Gymnasium中MuJoCo Half Cheetah的新基准，包含五个随机放置的障碍物，每个障碍物需对应不同行为。该基准作为MATPG的应用场景，验证其在连续型多任务环境中的可行性。实验表明，结合词典选择（lexicase selection）后，MATPG在多任务场景下表现更优。此外，对演化图的可解释性分析显示，模型的决策流程完全可读。
原文摘要 · Abstract (English)
Over the past few decades, machine learning has been widely used to learn complex tasks. Reinforcement Learning (RL), inspired by human behavior, is a great example, as it involves developing specific behaviours for specific tasks. To further challenge algorithms, Multi-Task RL (MTRL) environments have been introduced, requiring a single model to learn multiple behaviors. The Tangled Program Graph (TPG) algorithm is a Genetic Programming (GP) algorithm designed for discrete MTRL environments. Recently, the MAPLE algorithm has been proposed, as another GP algorithm that achieves high results in single task continuous RL environments. A variation of the TPG is proposed alongside MAPLE, named Multi-Action TPG (MATPG) that aggregates MAPLE agents, and creates a control flow to activate them. Initially tested on single task RL environments only, MATPG achieved similar results to MAPLE. In this work, we present a new benchmark based on the MuJoCo Half Cheetah from Gymnasium. This benchmark features five distinct obstacles that are randomly positioned in front of the agent, each of which demands a unique behavior. This benchmark serves as a use case for MATPG, to prove its ability as a GP solution for continuous MTRL environments. Our experiments demonstrate its superiority in this multi-task use case when combined with lexicase selection. Furthermore, we examine the interpretability of the evolved graph, revealing that the decision flow of the model is fully interpretable.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。