用贝尔曼误差最大化实现稳定确定性探索，优于传统ε-贪婪策略。
Deterministic Exploration via Stationary Bellman Error Maximization
- 以贝尔曼误差为目标构建确定性探索策略，避免随机噪声干扰。
- 在密集与稀疏奖励场景中均超越ε-贪婪，提升探索效率。
- 适合需要稳定探索的复杂强化学习任务，如高维环境或长序列决策。
探索是强化学习中关键且尚未解决的核心问题。现有方法多通过动作加噪、熵最大化或引入内在奖励来鼓励智能体进入状态空间的新区域。另一思路是将贝尔曼误差作为独立探索目标。本文提出三项改进：引入对利用状态感知的探索代理，使其能结合过往经验；设计对回合长度无关的目标函数；缓解远期策略学习带来的不稳定性。实验表明，该方法在密集和稀疏奖励环境下均优于ε-贪婪策略。
原文摘要 · Abstract (English)
Exploration is a crucial and distinctive aspect of reinforcement learning (RL) that remains a fundamental open problem. Several methods have been proposed to tackle this challenge. Commonly used methods inject random noise directly into the actions, indirectly via entropy maximization, or add intrinsic rewards that encourage the agent to steer to novel regions of the state space. Another previously seen idea is to use the Bellman error as a separate optimization objective for exploration. In this paper, we introduce three modifications to stabilize the latter and arrive at a deterministic exploration policy. Our separate exploration agent is informed about the state of the exploitation, thus enabling it to account for previous experiences. Further components are introduced to make the exploration objective agnostic toward the episode length and to mitigate instability introduced by far-off-policy learning. Our experimental results show that our approach can outperform $\varepsilon$-greedy in dense and sparse reward settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。