揭示强化学习与动态规划在决策优化中的本质联系
Relating Reinforcement Learning to Dynamic Programming-Based Planning
- 提出去随机化的强化学习框架，实现与动态规划算法的直接对比
- 证明代价最小化与奖励最大化等价的数学条件，揭示无限时长远期学习与单次目标达成的关联
- 强调应优化真实代价而非依赖人为设定的学习率等参数，适合研究决策理论的学者
本文弥合了最优规划与强化学习（RL）之间的部分差距，二者均源于动态规划在序列决策或最优控制中的应用。规划通常偏好确定性模型、单次目标终止和代价最小化，而强化学习则倾向于随机模型、无限时长远期折扣及奖励最大化，并引入学习率、贪婪度等学习相关参数。本文提出并分析了一种去随机化的强化学习方法，通过简单规划模型与值迭代和Dijkstra算法进行性能对比。数学分析表明：1）代价最小化与奖励最大化等价的条件；2）单次目标终止与无限时长远期学习等价的条件；3）折扣因子导致目标达成失败的条件。论文主张定义并优化真实代价，而非人为设定参数以引导操作。性能研究进一步扩展至随机情形，采用规划导向的标准，比较值迭代与带学习率和贪婪度的强化学习。
原文摘要 · Abstract (English)
This paper bridges some of the gap between optimal planning and reinforcement learning (RL), both of which share roots in dynamic programming applied to sequential decision making or optimal control. Whereas planning typically favors deterministic models, goal termination, and cost minimization, RL tends to favor stochastic models, infinite-horizon discounting, and reward maximization in addition to learning-related parameters such as the learning rate and greediness factor. A derandomized version of RL is developed, analyzed, and implemented to yield performance comparisons with value iteration and Dijkstra's algorithm using simple planning models. Next, mathematical analysis shows: 1) conditions under which cost minimization and reward maximization are equivalent, 2) conditions for equivalence of single-shot goal termination and infinite-horizon episodic learning, and 3) conditions under which discounting causes goal achievement to fail. The paper then advocates for defining and optimizing truecost, rather than inserting arbitrary parameters to guide operations. Performance studies are then extended to the stochastic case, using planning-oriented criteria and comparing value iteration to RL with learning rates and greediness factors.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。