Q-learning在连续环境中可能根本性不稳定，即使无模型误差也易失效。
Is Q-learning an Ill-posed Problem?
- 通过逐步消除误差源，验证了Q-learning本质上的不适定性
- 简单基准下仍出现学习失败，说明问题非由模型误差引发
- 对依赖Q-learning的从业者提出警示，尤其在复杂环境应用时
本文研究了Q-learning在连续环境中的不稳定性，这一问题在实践中频繁出现。传统上，这种不稳定性归因于自举（bootstrapping）和回归模型误差。我们使用一个典型的强化学习基准，通过逐步消除这些潜在误差源，系统评估了自举与模型不准确的影响。结果表明，即使在相对简单的基准中，Q-learning的核心任务——从策略相关的目标值迭代学习Q函数——也可能本质上是不适定的，容易导致失败。这些发现质疑了Q-learning作为强化学习通用解决方案的可靠性。
原文摘要 · Abstract (English)
This paper investigates the instability of Q-learning in continuous environments, a challenge frequently encountered by practitioners. Traditionally, this instability is attributed to bootstrapping and regression model errors. Using a representative reinforcement learning benchmark, we systematically examine the effects of bootstrapping and model inaccuracies by incrementally eliminating these potential error sources. Our findings reveal that even in relatively simple benchmarks, the fundamental task of Q-learning - iteratively learning a Q-function from policy-specific target values - can be inherently ill-posed and prone to failure. These insights cast doubt on the reliability of Q-learning as a universal solution for reinforcement learning problems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。