在线性函数逼近下，为非马尔可夫过程设计了收敛的强化学习算法。
Reinforcement Learning with Function Approximation for Non-Markov Processes
- 基于正交投影与辅助马尔可夫过程贝尔曼算子的联合算子进行策略评估
- 在特定基函数（量化映射生成）下，Q-learning可保证收敛
- 适用于部分可观测马尔可夫决策过程，给出显式误差界
研究在非马尔可夫状态和代价过程下使用线性函数逼近的强化学习方法。首先分析策略评估方法，在底层非马尔可夫过程满足合适遍历性条件下，证明算法收敛；其极限对应于一个由正交投影与辅助马尔可夫决策过程贝尔曼算子组成的联合算子的不动点。对于带线性函数逼近的Q-learning，如在马尔可夫情形一般，通常不保证收敛；但当基函数由量化映射构造时，可在类似遍历性条件下证明收敛。最后将结果应用于部分可观测马尔可夫决策过程，采用有限记忆变量作为状态表示，并推导出相应学习算法极限的显式误差界。
原文摘要 · Abstract (English)
We study reinforcement learning methods with linear function approximation under non-Markov state and cost processes. We first consider the policy evaluation method and show that the algorithm converges under suitable ergodicity conditions on the underlying non-Markov processes. Furthermore, we show that the limit corresponds to the fixed point of a joint operator composed of an orthogonal projection and the Bellman operator of an auxiliary \emph{Markov} decision process. For Q-learning with linear function approximation, as in the Markov setting, convergence is not guaranteed in general. We show, however, that for the special case where the basis functions are chosen based on quantization maps, the convergence can be shown under similar ergodicity conditions. Finally, we apply our results to partially observed Markov decision processes, where finite-memory variables are used as state representations, and we derive explicit error bounds for the limits of the resulting learning algorithms.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。