用内在好奇心驱动探索，解决稀疏奖励下的强化学习难题
Endogenous Exploration in Reinforcement Learning with Intrinsic Curiosity
- 结合外部奖励与认知动机，引导智能体有结构地探索
- 在月球着陆和双足行走任务中表现媲美PPO和ICM
- 揭示中等混乱度最利于有效探索，适合动态环境研究
我们提出一种强化学习框架，通过内在好奇心驱动探索，适用于环境非平稳且奖励稀疏、延迟、无信息或缺失的场景。智能体的动作选择由外部奖励与一种认知动机机制共同决定，该机制促使智能体偏向有结构的探索方向。核心假设是：在中等混乱度下探索效果最佳，而过度僵化或过于紊乱的动态都会导致性能下降。我们在液态状态机（LSM）基础上实现该框架，并在两个标准基准上评估：离散动作的LunarLanderv2和连续控制的BipedalWalkerv3。结果表明，该方法在两项任务上均达到与PPO和内在好奇心模块（ICM）相当的性能。进一步分析显示，主动推理代理在相同测试下未恢复好奇心窗口，说明所提动力学捕捉了独特的探索模式。
原文摘要 · Abstract (English)
We propose a reinforcement learning framework in which exploration is driven by intrinsic curiosity, designed for scenarios where environments are non-stationary and rewards are sparse, delayed, uninformative, or absent. In our model, action selection is guided by a combination of external rewards and an epistemic motivation mechanism that biases the agent toward structured exploratory directions. The central hypothesis is that effective exploration emerges at intermediate levels of incoherence, while performance degrades under both overly rigid and overly disordered dynamics. To test this idea, we implement the framework on top of a Liquid State Machine (LSM) substrate and evaluate it on two standard benchmarks: the discrete-action LunarLanderv2 and the continuous-control BipedalWalkerv3. The proposed method achieves competitive performance on both tasks relative to established deep RL algorithms, including Proximal Policy Optimization (PPO) and Intrinsic Curiosity Module (ICM). We further show that the curiosity window is not recovered in Active Inference agents under the same analysis, suggesting that the proposed dynamics capture a distinct exploration regime
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。