arXiv:2503.01885cs.LGcs.AI2025-03ICML被引 5

为多样化任务设计可保证性能的策略委员会,提升强化学习泛化能力。

Learning Policy Committees for Effective Personalization in MDPs with Diverse Tasks

  • 构建策略委员会,确保执行时近似最优策略存在
  • 低维任务下有理论保证,高维任务用梯度优化实用方案
  • 适合需要少样本快速适应的机器人控制等场景

许多动态决策问题(如机器人控制)涉及一系列任务,其中许多任务在训练时未知。传统多任务与元强化学习方法在任务多样性高时泛化能力差。现有处理任务多样性的方法(如任务嵌入作策略上下文、任务聚类)通常缺乏性能保证且需大量训练任务。为此,我们提出一种新方法:学习一个策略委员会,使得在执行中遇到的任务至少存在一个近似最优策略的概率很高。尽管该问题一般不可近似,我们提出了两种实用算法:第一种在任务为低维时提供可证明的近似与采样复杂度保证;第二种为通用且实用的基于梯度的方法。此外,我们还提供了少样本学习的可证明样本复杂度边界。在MuJoCo和Meta-World上的实验表明,所提方法在训练、泛化和少样本学习方面均显著优于当前最先进的多任务、元强化学习及任务聚类基线,性能差距常达显著水平。代码已开源:https://github.com/CERL-WUSTL/PACMAN。

原文摘要 · Abstract (English)

Many dynamic decision problems, such as robotic control, involve a series of tasks, many of which are unknown at training time. Typical approaches for these problems, such as multi-task and meta reinforcement learning, do not generalize well when the tasks are diverse. On the other hand, approaches that aim to tackle task diversity, such as using task embedding as policy context and task clustering, typically lack performance guarantees and require a large number of training tasks. To address these challenges, we propose a novel approach for learning a policy committee that includes at least one near-optimal policy with high probability for tasks encountered during execution. While we show that this problem is in general inapproximable, we present two practical algorithmic solutions. The first yields provable approximation and task sample complexity guarantees when tasks are low-dimensional (the best we can do due to inapproximability), whereas the second is a general and practical gradient-based approach. In addition, we provide a provable sample complexity bound for few-shot learning. Our experiments on MuJoCo and Meta-World show that the proposed approach outperforms state-of-the-art multi-task, meta-, and task clustering baselines in training, generalization, and few-shot learning, often by a large margin. Our code is available at https://github.com/CERL-WUSTL/PACMAN.

强化学习策略委员会少样本学习任务多样性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。