提出自动探索方法，让强化学习在无参数情况下高效探索状态与动作空间。
Auto-exploration for online reinforcement learning
- 通过自适应探索机制，无需预设参数即可覆盖状态和动作空间。
- 在表格和线性近似设置下均达到O(ε⁻²)样本复杂度，优于以往依赖算法参数的方法。
- 适合追求高效、无调参强化学习的工程与研究者使用。
强化学习中的探索-利用权衡是高效算法的核心挑战。现有有限状态与动作的折扣强化学习算法依赖对状态和动作空间的充分探索，但导致不可实现且性能次优。为此，本文提出一类新型自动探索方法，可同时自动探索状态与动作空间，适用于表格型与线性函数逼近场景。在不依赖算法特定假设的前提下，两种设置均实现解决ε误差所需的O(ε⁻²)样本复杂度，该结果新颖，避免了先前工作中可能出现任意大的算法依赖参数。方法简单易实现，因其完全无参数。通过将自动探索融入策略镜像下降，规避了以往方法中未知的平稳分布问题。在表格设置中引入基于数据驱动的动态探索停止时间；在线性函数逼近中，提出一种基于折扣访问分布的新采样策略，可覆盖更广泛的马尔可夫链类。
原文摘要 · Abstract (English)
The exploration-exploitation dilemma in reinforcement learning (RL) is a fundamental challenge to efficient RL algorithms. Existing algorithms for finite state and action discounted RL problems address this by assuming sufficient exploration over both state and action spaces. However, this yields non-implementable algorithms and sub-optimal performance. To resolve these limitations, we introduce a new class of methods with auto-exploration, or methods that automatically explore both state and action spaces. Auto-exploration can be applied in both the tabular and linear function approximation setting. Under algorithm-independent assumptions on the existence of an exploring optimal policy, both settings attain $O(ε^{-2})$ sample complexity to solve to $ε$ error. These complexities are novel since they avoid algorithm-dependent parameters seen in prior works, which may be arbitrarily large. The methods are also simple to implement because they are parameter-free. We achieve these results by integrating auto-exploration into policy mirror descent to avoid the (unknown) stationary distribution seen in prior art. In the tabular setting, we introduce a dynamic exploration time with a data-driven stopping time, while for linear function approximation we propose a new sampling distribution based on the discounted visitation distribution that covers a more general class of Markov chains.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。