让智能体从成功与失败中学习,自适应调整行为。
Autonomous Learning From Success and Failure: Goal-Conditioned Supervised Learning with Negative Feedback
- 用正负反馈双重评估轨迹,兼顾成功与失败经验。
- 无需预设奖励函数,通过对比学习自动构建纠错目标。
- 适合目标动态变化的长期自适应系统,如机器人自主学习。
从奖励函数和示范学习是训练自主系统的主要方法,但都需要人为指定每种行为,难以应对长期运行中目标与环境未知的情况。近期提出的基于自模仿的目标条件监督学习(GCSL)通过重标注目标实现自监督,但存在两个问题:仅依赖自身经验会放大固有偏差;重标注策略只关注成功结果,忽略失败教训。为此,本文提出带负反馈的目标条件监督学习(GCSL-NF),对每条轨迹进行两次评估:一次按重标注目标正向评价,一次按原始目标进行纠正性评价。纠正目标由从轨迹邻域关系中对比学习得到的相似性函数决定,无需定义奖励函数或几何距离。实验表明,GCSL-NF克服了初始偏差,随学习进程更有效利用负反馈,性能优于或媲美传统GCSL与HER方法。该方法降低了对预设奖励函数的依赖,特别适用于目标多样、动态或难以工程化的自适应自主系统。
原文摘要 · Abstract (English)
Learning from reward functions and imitation learning of demonstrations are the two principal approaches for training autonomous systems that interact with an environment through action and observation. Both, however, require human specification for each behaviour to be acquired, a problem for long-lived self-adaptive systems whose goals and operating conditions cannot be fully anticipated at design time. Recently, Goal-Conditioned Supervised Learning (GCSL) through self-imitation has been proposed as a self-supervised alternative: by strategically relabelling goals, agents can derive policy insights from their own experiences. Despite its successes, this framework presents two notable limitations: (1) learning exclusively from self-generated experiences can exacerbate the agents' inherent biases; (2) the relabelling strategy allows agents to focus solely on successful outcomes, precluding them from learning from their mistakes. To address these issues, we propose GCSL with Negative Feedback (GCSL-NF), which evaluates each trajectory twice: positively with respect to relabelled goals, and correctively with respect to the goal originally intended. The corrective target comes from a similarity function learned contrastively from trajectory-induced neighbourhood relations, so that neither a reward function nor a geometric distance needs to be specified. Our experiments show that GCSL-NF overcomes limitations imposed by agents' initial biases, increasingly benefits from negative feedback as learning progresses, and matches or surpasses GCSL- and HER-based methods. By reducing reliance on prespecified reward functions, the proposed approach is particularly relevant for self-adaptive autonomous systems, where adaptation objectives may be diverse, changing, or difficult to engineer.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。