用生物启发的惊喜机制，让AI在奖励稀疏时学得更快更稳。
Deep Intrinsic Surprise-Regularized Control (DISRC): A Biologically Inspired Mechanism for Efficient Deep Q-Learning in Sparse Environments
- 根据隐空间中的意外程度动态调节学习强度，增强探索期的灵活性。
- 在门钥匙任务中比传统DQN快33%达成首次成功，奖励波动更小。
- 适合奖励稀疏、反馈延迟的强化学习场景，如机器人控制与复杂决策。
深度强化学习在自主控制中取得重大进展，但标准DQN代理依赖固定学习率和均匀更新尺度，即使更新受时序差分（TD）误差调节，这种僵化性仍会破坏收敛，尤其在奖励稀疏环境中。本文提出生物启发的深度内在惊喜正则化控制（DISRC），通过隐空间惊喜度动态调整Q值更新。DISRC采用基于层归一化的编码器，计算状态相对于移动隐空间基准点的偏差型惊喜分数，更新强度同时依赖于TD误差与惊喜程度，促进早期探索的可塑性，随熟悉度提升实现稳定性。在两个稀疏奖励的MiniGrid环境（MiniGrid-DoorKey-8x8 和 MiniGrid-LavaCrossingS9N1）上评估，与基线DQN相同设置下，门钥匙任务中DISRC提前33%完成首次成功（79对118轮），奖励标准差更低（0.25对0.34），奖励曲线下面积（AUC）更高（596.42对534.90）。在熔岩穿越任务中，最终奖励更高（0.95对0.93），且达到最高AUC（957.04），尽管收敛较慢。结果表明，DISRC是一种新颖的离策略智能体学习强度调控机制，显著提升稀疏奖励场景下的效率与稳定性。
原文摘要 · Abstract (English)
Deep reinforcement learning (DRL) has driven major advances in autonomous control. Still, standard Deep Q-Network (DQN) agents tend to rely on fixed learning rates and uniform update scaling, even as updates are modulated by temporal-difference (TD) error. This rigidity destabilizes convergence, especially in sparse-reward settings where feedback is infrequent. We introduce Deep Intrinsic Surprise-Regularized Control (DISRC), a biologically inspired augmentation to DQN that dynamically scales Q-updates based on latent-space surprise. DISRC encodes states via a LayerNorm-based encoder and computes a deviation-based surprise score relative to a moving latent setpoint. Each update is then scaled in proportion to both TD error and surprise intensity, promoting plasticity during early exploration and stability as familiarity increases. We evaluate DISRC on two sparse-reward MiniGrid environments, which included MiniGrid-DoorKey-8x8 and MiniGrid-LavaCrossingS9N1, under identical settings as a vanilla DQN baseline. In DoorKey, DISRC reached the first successful episode (reward > 0.8) 33% faster than the vanilla DQN baseline (79 vs. 118 episodes), with lower reward standard deviation (0.25 vs. 0.34) and higher reward area under the curve (AUC: 596.42 vs. 534.90). These metrics reflect faster, more consistent learning - critical for sparse, delayed reward settings. In LavaCrossing, DISRC achieved a higher final reward (0.95 vs. 0.93) and the highest AUC of all agents (957.04), though it converged more gradually. These preliminary results establish DISRC as a novel mechanism for regulating learning intensity in off-policy agents, improving both efficiency and stability in sparse-reward domains. By treating surprise as an intrinsic learning signal, DISRC enables agents to modulate updates based on expectation violations, enhancing decision quality when conventional value-based methods fall short.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。