用无标签动作数据训练机器人,实现零样本控制。
Zero-Shot Whole-Body Humanoid Control via Behavioral Foundation Models
- 通过正向反向表示对齐动作数据与状态空间,引导策略模仿真实行为。
- 在仅观察动作捕捉数据下,实现多任务零样本控制,性能媲美专用模型。
- 适合需要快速部署、无需额外训练的机器人控制场景。
无监督强化学习旨在预训练能解决复杂环境中多种下游任务的智能体。然而现有方法存在诸多局限:可能需针对每个下游任务运行强化学习过程以达到满意性能,或依赖覆盖全面、标注良好的任务特定数据集,或使用与目标任务相关性差的无监督损失函数进行策略预训练。本文提出一种新算法,通过模仿未标注行为数据中的轨迹来正则化无监督强化学习。核心创新在于构建前向-后向表示,将未标注轨迹嵌入与状态、奖励和策略相同的潜在空间,并利用潜变量条件判别器促使策略覆盖行为数据中的状态分布。由此可学习到与数据中行为高度一致、同时保持零样本泛化能力的策略。我们在具有挑战性的人形机器人控制任务中验证了该方法的有效性:仅使用仅观测的动作捕捉数据,训练出首个可提示式操作的人形行为基础模型 Meta Motivo,能够完成运动追踪、目标到达和奖励优化等多种全身任务。该模型展现出类人行为特征,在多项任务上性能优于当前最先进的无监督强化学习与基于模型的基线方法。
原文摘要 · Abstract (English)
Unsupervised reinforcement learning (RL) aims at pre-training agents that can solve a wide range of downstream tasks in complex environments. Despite recent advancements, existing approaches suffer from several limitations: they may require running an RL process on each downstream task to achieve a satisfactory performance, they may need access to datasets with good coverage or well-curated task-specific samples, or they may pre-train policies with unsupervised losses that are poorly correlated with the downstream tasks of interest. In this paper, we introduce a novel algorithm regularizing unsupervised RL towards imitating trajectories from unlabeled behavior datasets. The key technical novelty of our method, called Forward-Backward Representations with Conditional-Policy Regularization, is to train forward-backward representations to embed the unlabeled trajectories to the same latent space used to represent states, rewards, and policies, and use a latent-conditional discriminator to encourage policies to ``cover'' the states in the unlabeled behavior dataset. As a result, we can learn policies that are well aligned with the behaviors in the dataset, while retaining zero-shot generalization capabilities for reward-based and imitation tasks. We demonstrate the effectiveness of this new approach in a challenging humanoid control problem: leveraging observation-only motion capture datasets, we train Meta Motivo, the first humanoid behavioral foundation model that can be prompted to solve a variety of whole-body tasks, including motion tracking, goal reaching, and reward optimization. The resulting model is capable of expressing human-like behaviors and it achieves competitive performance with task-specific methods while outperforming state-of-the-art unsupervised RL and model-based baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。