通过监测学习进度，让智能体避开无意义的随机干扰，更高效探索。
Beyond Noisy-TVs: Noise-Robust Exploration Via Learning Progress Monitoring
- 用模型改进程度替代预测误差作为内在奖励，引导智能体关注可学变化。
- 在3D迷宫中探索状态数提升40%，Atari游戏外在奖励提高27%。
- 适合需要抗干扰探索的强化学习场景，如机器人导航或复杂环境建模。
当环境中存在不可学习的随机源（如噪声电视）时，传统基于内在奖励的探索方法会陷入该源并停止有效探索。基于不确定性估计或分布相似性的方法虽能随时间逃离噪声源，但样本效率低且计算开销大。受神经科学启发，我们提出学习进度监控（LPM）机制：在探索过程中，奖励模型改进而非预测误差或新颖性，从而有效鼓励智能体关注可学习的转移。LPM采用双网络设计，利用误差模型预测动态模型前一迭代的预期预测误差，并以当前与前一迭代的误差差值指导探索。理论上证明，LPM的内在奖励具有零等变性，且是信息增益（IG）的单调指标，而误差模型对实现与IG的单调对应至关重要。在基于MNIST、160x120 RGB输入的3D迷宫及Atari游戏的噪声环境下，实验表明LPM的内在奖励收敛更快，在迷宫中探索状态更多，Atari上外在奖励更高。这一概念简单的新方法标志着抗噪声探索范式的转变。
原文摘要 · Abstract (English)
When there exists an unlearnable source of randomness (noisy-TV) in the environment, a naively intrinsic reward driven exploring agent gets stuck at that source of randomness and fails at exploration. Intrinsic reward based on uncertainty estimation or distribution similarity, while eventually escapes noisy-TVs as time unfolds, suffers from poor sample efficiency and high computational cost. Inspired by recent findings from neuroscience that humans monitor their improvements during exploration, we propose a novel method for intrinsically-motivated exploration, named Learning Progress Monitoring (LPM). During exploration, LPM rewards model improvements instead of prediction error or novelty, effectively rewards the agent for observing learnable transitions rather than the unlearnable transitions. We introduce a dual-network design that uses an error model to predict the expected prediction error of the dynamics model in its previous iteration, and use the difference between the model errors of the current iteration and previous iteration to guide exploration. We theoretically show that the intrinsic reward of LPM is zero-equivariant and a monotone indicator of Information Gain (IG), and that the error model is necessary to achieve monotonicity correspondence with IG. We empirically compared LPM against state-of-the-art baselines in noisy environments based on MNIST, 3D maze with 160x120 RGB inputs, and Atari. Results show that LPM's intrinsic reward converges faster, explores more states in the maze experiment, and achieves higher extrinsic reward in Atari. This conceptually simple approach marks a shift-of-paradigm of noise-robust exploration. For code to reproduce our experiments, see https://github.com/Akuna23Matata/LPM_exploration
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。