让智能体通过观察他人学习,无需访问其策略或奖励函数。
Intrinsic Vicarious Conditioning for Deep Reinforcement Learning

- 基于心理与生物机制,设计四步拟态学习流程:注意、记忆、复现、强化。
- 在迷你世界与CarRacing环境中,显著延长了智能体存活时间。
- 适用于单次生命学习或持续学习场景,支持低样本量训练。
强化学习中复杂的内在驱动力大多依赖直接条件化范式,限制了智能体从环境或他人中学习的能力。现有离策略或示范学习方法需获取示范者的策略或奖励函数。本文提出拟态条件化作为内在奖励机制,借鉴心理学与生物学研究,通过记忆方法实现注意、保留、复现与强化四个步骤。该机制支持低样本学习,无需示范者策略或奖励函数。我们在MiniWorld Sidewalk(无死亡奖励的非描述性终止条件)和Box2D CarRacing环境中验证,结果表明拟态条件化能有效避免非描述性终止状态,引导智能体向理想状态迁移,显著延长单次试运行时长。整体上,该方法更贴近认知可解释的学习范式,适用于单次生命学习或持续学习任务。
原文摘要 · Abstract (English)
Advancements in reinforcement learning have produced a variety of complex and useful intrinsic driving forces; crucially, these drivers operate under a direct conditioning paradigm. This form of conditioning limits our agents' capacity by restricting how they learn from the environment as well as from others. Off-policy or learn-by-example methods can learn from demonstrators' representations, but they require access to the demonstrating agent's policies or their reward functions. Our work overcomes this direct sampling limitation by introducing vicarious conditioning as an intrinsic reward mechanism. We draw from psychological and biological literature to provide a foundation for vicarious conditioning and use memory-based methods to implement its four steps: attention, retention, reproduction, and reinforcement. Crucially, our vicarious conditioning paradigms support low-shot learning and do not require the demonstrator agent's policy nor its reward functions. We evaluate our approach in the MiniWorld Sidewalk environment, one of the few public environments that features a non-descriptive terminal condition (no reward provided upon agent death), and extend it to Box2D's CarRacing environment. Our results across both environments demonstrate that vicarious conditioning enables longer episode lengths by discouraging the agent from non-descriptive terminal conditions and guiding the agent toward desirable states. Overall, this work emulates a cognitively-plausible learning paradigm better suited to problems such as single-life learning or continual learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。