提出新算法，让强化学习在非马尔可夫环境下更稳定收敛。
Commit to the Bit: Reactive Reinforcement Learning Done Right
- 行为策略进入特征后固定动作，仅当特征变化才重采样。
- 在确定性观测下几乎必然收敛到最优反应策略。
- 比以往方法更宽松的条件，适合复杂真实场景。
强化学习算法通常在马尔可夫假设下分析（或设计），但实际环境多为部分可观测，或需函数逼近导致代理无法访问马尔可夫状态特征。本文研究在具有确定性观测（或等价于硬状态聚合）的有限环境中学习最优反应策略的问题。提出一种新算法——已承诺Q-learning，证明在称为‘重连鲁棒性’的直观假设下，几乎必然收敛至最优反应策略。该假设严格弱于以往工作中使用的$ q_\star $-可实现性条件。该算法是经典Q-learning的变体：代理进入某一特征后即承诺单一动作，仅当观察到的特征发生变化时才重新采样动作。分析中的关键在于引入准马尔可夫环境。
原文摘要 · Abstract (English)
Reinforcement learning algorithms are commonly analyzed (and designed) under the Markov assumption. This is unrealistic, as most environments encountered in practice are either partially observable, or require function approximation that restricts the agent to access non-Markovian state features. We consider the problem of learning an optimal reactive policy in a finite environment with deterministic observations (or equivalently, hard state aggregation). We introduce a new algorithm, Committed Q-learning, and prove almost-sure convergence to the optimal reactive policy under an intuitive assumption we call rewire-robustness. This assumption is strictly weaker than the $q_\star$-realizability condition used in prior work. Our algorithm is a variant of classical Q-learning in which the behavior policy commits to a single action upon entering a feature, and only resamples actions when the observed feature changes. A crucial part of our analysis is the introduction of quasi-Markov environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。