用专家视频自动学奖励,让机器人持续学习不遗忘。
ProgAgent:A Continual RL Agent with Progress-Aware Rewards
- 从无标注视频中提取任务进展,生成密集奖励信号
- 在连续学习中减少遗忘,比基线快2.3倍,接近理想记忆表现
- 适合需要长期学习的机器人系统,尤其擅长少样本复杂操作
我们提出ProgAgent,一种结合进度感知奖励学习与高效JAX原生架构的持续强化学习(CRL)代理。长期机器人学习面临灾难性遗忘和奖励标注成本高的问题。ProgAgent通过感知模型,从无标签专家视频中估计初始、当前与目标状态的任务进展,生成密集且结构化的奖励信号。理论上可解释为学习到的状态势函数,引导行为符合专家模式。为应对在线探索中出现的新颖分布外状态,引入对抗性回推优化,抑制奖励模型对非专家轨迹的过度自信预测,缓解分布偏移。将该奖励机制嵌入即时编译循环,支持大规模并行采样与全可微更新,实现统一优化目标:融合PPO、核心集回放与突触智能,提升稳定-可塑性平衡。在ContinualBench与Meta-World基准测试中,ProgAgent显著减少遗忘,加速学习,优于关键基线(如Rank2Reward、TCN、Coreset、SI),甚至超越理想化完美记忆代理。真实机器人实验验证其能从噪声多、样本少的人类示范中习得复杂操纵技能。
原文摘要 · Abstract (English)
We present ProgAgent, a continual reinforcement learning (CRL) agent that unifies progress-aware reward learning with a high-throughput, JAX-native system architecture. Lifelong robotic learning grapples with catastrophic forgetting and the high cost of reward specification. ProgAgent tackles these by deriving dense, shaped rewards from unlabeled expert videos through a perceptual model that estimates task progress across initial, current, and goal observations. We theoretically interpret this as a learned state-potential function, delivering robust guidance in line with expert behaviors. To maintain stability amid online exploration - where novel, out-of-distribution states arise - we incorporate an adversarial push-back refinement that regularizes the reward model, curbing overconfident predictions on non-expert trajectories and countering distribution shift. By embedding this reward mechanism into a JIT-compiled loop, ProgAgent supports massively parallel rollouts and fully differentiable updates, rendering a sophisticated unified objective feasible: it merges PPO with coreset replay and synaptic intelligence for an enhanced stability-plasticity balance. Evaluations on ContinualBench and Meta-World benchmarks highlight ProgAgent's advantages: it markedly reduces forgetting, boosts learning speed, and outperforms key baselines in visual reward learning (e.g., Rank2Reward, TCN) and continual learning (e.g., Coreset, SI) - surpassing even an idealized perfect memory agent. Real-robot trials further validate its ability to acquire complex manipulation skills from noisy, few-shot human demonstrations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。