arXiv:2411.01146cs.LGcs.AI2024-11被引 1

提出可自适应匹配任务的多任务强化学习模型,无需事先知道任务类型。

Task-Aware Harmony Multi-Task Decision Transformer for Offline Reinforcement Learning

  • 通过双层优化自动发现每项任务的最佳参数子空间。
  • 在已知任务下提升8%,未知任务下提升10%。
  • 不依赖任务标识符,适合真实复杂场景应用。

离线多任务强化学习的目标是构建一个适用于多种任务的统一策略,无需在线环境交互。现有方法采用序列建模,利用Transformer架构的可扩展性与参数共享优势来挖掘任务相似性。然而,任务内容和复杂度差异带来挑战,需合理分配参数并处理冲突梯度。此外,通常需要推理时已知任务标识符,限制了在任务未知的真实场景中的应用。本文提出和谐多任务决策变压器(HarmoDT),通过元学习框架将问题建模为双层优化:上层学习掩码以定义各任务的和谐参数子空间,下层优化参数以提升统一策略性能。为消除对任务标识符的需求,进一步设计分组变体G-HarmoDT,基于梯度信息聚类任务,并用门控网络推断任务类别。在多个基准测试中验证表明,该方法在提供任务设置下提升8%,任务无关设置下提升5%,未见任务设置下提升10%。

原文摘要 · Abstract (English)

The purpose of offline multi-task reinforcement learning (MTRL) is to develop a unified policy applicable to diverse tasks without the need for online environmental interaction. Recent advancements approach this through sequence modeling, leveraging the Transformer architecture's scalability and the benefits of parameter sharing to exploit task similarities. However, variations in task content and complexity pose significant challenges in policy formulation, necessitating judicious parameter sharing and management of conflicting gradients for optimal policy performance. Furthermore, identifying the optimal parameter subspace for each task often necessitates prior knowledge of the task identifier during inference, limiting applicability in real-world scenarios with variable task content and unknown current tasks. In this work, we introduce the Harmony Multi-Task Decision Transformer (HarmoDT), a novel solution designed to identify an optimal harmony subspace of parameters for each task. We formulate this as a bi-level optimization problem within a meta-learning framework, where the upper level learns masks to define the harmony subspace, while the inner level focuses on updating parameters to improve the overall performance of the unified policy. To eliminate the need for task identifiers, we further design a group-wise variant (G-HarmoDT) that clusters tasks into coherent groups based on gradient information, and utilizes a gating network to determine task identifiers during inference. Empirical evaluations across various benchmarks highlight the superiority of our approach, demonstrating its effectiveness in the multi-task context with specific improvements of 8% gain in task-provided settings, 5% in task-agnostic settings, and 10% in unseen settings.

多任务强化学习决策变压器离线学习自适应参数

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。