提出一种无需调参的强化学习探索方法，让智能体更稳定高效地试错。
Hyper: Hyperparameter Robust Efficient Exploration in Reinforcement Learning
- 通过正则化探索访问频次，解耦利用与探索，提升训练稳定性。
- 在多种环境上表现稳健，无需额外超参数调整即可达到优异性能。
- 理论证明在函数逼近下具有有效性，适合追求鲁棒性的研究者使用。
探索与利用的权衡是强化学习中的核心挑战。近年来，基于好奇心的探索方法在解决高难度探索问题上取得了显著成效，但其在不同环境中均需大量超参数调优，严重限制了该类方法的适用性与可及性。本文通过分析智能体行为，揭示了选择合适超参数的根本困难，并指出在好奇心驱动学习时优化过程存在不稳定问题。为此，我们提出超参数鲁棒探索（Hyper），通过有效正则化探索的访问频次并解耦利用机制，显著缓解上述问题。理论上，我们证明了Hyper在函数逼近设置下的有效性；实验上，其在多种环境中展现出卓越的性能与鲁棒性。
原文摘要 · Abstract (English)
The exploration \& exploitation dilemma poses significant challenges in reinforcement learning (RL). Recently, curiosity-based exploration methods achieved great success in tackling hard-exploration problems. However, they necessitate extensive hyperparameter tuning on different environments, which heavily limits the applicability and accessibility of this line of methods. In this paper, we characterize this problem via analysis of the agent behavior, concluding the fundamental difficulty of choosing a proper hyperparameter. We then identify the difficulty and the instability of the optimization when the agent learns with curiosity. We propose our method, hyperparameter robust exploration (\textbf{Hyper}), which extensively mitigates the problem by effectively regularizing the visitation of the exploration and decoupling the exploitation to ensure stable training. We theoretically justify that \textbf{Hyper} is provably efficient under function approximation setting and empirically demonstrate its appealing performance and robustness in various environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。