arXiv:2510.17059cs.LG2025-10被引 1

用目标推理实现无需训练的零样本模仿,让机器人一眼看懂示范动作。

Consistent Zero-Shot Imitation with Contrastive Goal Inference

  • 通过对比强化学习自动推断目标状态,把复杂任务简化为找目标。
  • 在无奖励、无演示情况下预训练,测试时直接复现专家行为。
  • 适合需要快速部署的机器人控制场景,尤其擅长理解抽象动作意图。

零样本模仿学习要求智能体仅凭一次示范即可复现专家行为,且测试时无需环境交互或梯度更新。本文提出对比逆强化学习(CIRL),一种用于预训练零样本模仿智能体的自监督框架。核心思想是:许多任务可由单一目标状态概括。因此可将多任务逆强化学习转化为更易处理的目标推断问题,并利用先进目标条件强化学习方法恢复可达目标的策略。预训练阶段,CIRL联合使用三项组件:(1) 一种设计用于学习最大熵目标条件策略的对比强化学习变体;(2) 自动目标生成机制(GoalKDE),驱动探索;(3) 均值场变分模型,从轨迹中进行摊销式目标推断。我们证明该过程能一致地还原示范者的意图,考虑不同状态的可达难度;并说明结构相似的先前方法可能因忽略难度差异而推断错误奖励。在目标条件和标准奖励最大化控制任务上的实验表明,CIRL优于现有零样本模仿方法,验证了目标作为行为紧凑摘要的表达力。

原文摘要 · Abstract (English)

Zero-shot imitation learning requires an agent to reproduce expert behavior from a single demonstration without additional environment interaction or gradient updates at test time. We introduce Contrastive Inverse Reinforcement Learning (CIRL), a self-supervised framework for pre-training zero-shot imitation agents. Our methods rests on a key observation that many useful tasks can be summarized by a single goal state. We can thus convert the multi-task inverse RL problem into a more tractable goal-inference problem, and utilize state-of-the-art goal-conditioned RL methods to recover a policy that reaches the goal. During pre-training, CIRL jointly employs three components to learn without any rewards or demonstrations: (1) a variant of contrastive RL designed to learn maximum-entropy goal-conditioned policies, (2) an automatic goal proposal mechanism (GoalKDE) that drives exploration, and (3) a mean-field variational model that performs amortized goal inference from trajectories. We prove that this procedure consistently recovers the demonstrator's intent by accounting for the relative difficulty of reaching different states and show how structurally similar prior work may otherwise fail to infer the correct reward. Experiments on goal-conditioned and standard reward-maximizing control tasks show that CIRL outperforms prior zero-shot imitation methods, supporting the expressiveness of goals as a compact summary of behavior.

零样本模仿目标推理自监督学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。