SCOUT让每个任务场景自适应调整辅助训练,提升稀疏奖励强化学习效果。
SCOUT: Per-Context Reset Curricula for Sparse-Reward Reinforcement Learning
- 为不同任务场景动态分配辅助起点,按学习进度自主控制帮助移除
- 在6个场景中使原本失败的任务成功,且在冲突节奏下仍保持稳定性能
- 无需分组标签或修改模型,适合复杂多变的强化学习任务
稀疏奖励强化学习常因初始随机采样难以到达任务后期阶段而失败。重置课程通过从较易的中间状态(支架)开始训练来解决此问题。这类课程面临两个决策:支架访问(获取有效起始点)和支架分配(决定辅助移除速度)。以往方法采用统一调度,但在不同任务实例(上下文)学习速度不同时会失效。本文提出SCOUT,一种在线、与学习器无关的重置控制器,为每个上下文提供独立课程。仅使用二值化成功信号,当持续成功时移除辅助,失败后恢复,并在停滞时谨慎尝试更难起点,不改变奖励函数、优化器或学习算法。计数构造证明:同步全局调度可能不足,当上下文需要不同量的辅助练习时。在六个导航与操作任务中,支架访问显著提升学习表现,使三个原无法成功的任务达成目标。在构造的节奏冲突实验中,每种全局调度均导致一组未解,而SCOUT同时解决两组。平均成功率可能掩盖此失败,故报告最差组表现。基于群体的调度仅在学习差异已知分组时有效,但在组内出现差异时失效;而SCOUT无需分组标签,在两种情况下均保持一致优势。重置课程应按学习进度差异尺度移除辅助。
原文摘要 · Abstract (English)
Sparse-reward reinforcement learning often fails because rollouts from the unassisted evaluation start rarely reach later task stages. Reset curricula address this by starting some training rollouts from easier intermediate states, called scaffolds. Such a curriculum faces two decisions: scaffold access, obtaining informative starts, and scaffold allocation, deciding how quickly that assistance is removed. Most prior curricula pace removal on one shared schedule, which can fail when task instances, or contexts, learn at different rates. We introduce SCOUT, an online, learner-agnostic reset controller that gives every context its own curriculum. Using only binary rollout success, SCOUT removes assistance after sustained success, restores it after failure, and cautiously tests a harder start when progress stalls, without changing the reward, optimizer, or learner. A counting construction shows that synchronized global pacing can be insufficient when contexts need conflicting amounts of assisted practice. Across six navigation and manipulation settings, scaffold access improves learning and enables success in three where unassisted training fails within the reported budget. In a constructed pacing conflict, each tested global schedule leaves one group unsolved, while SCOUT solves both. Average success can conceal this failure, so we also report the least successful group. Group-level pacing works when learning differences follow known groups but can fail when they occur within one group. SCOUT needs no group labels and remains consistently strong in both cases. A reset curriculum should remove assistance at the scale where learning progress differs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。