提出隐式Q学习与SARSA，自动调节步长，大幅降低调参难度。
Implicit Q-Learning and SARSA: Liberating Policy Control from Step-Size Calibration
- 将更新规则转为不动点方程，步长自适应反比于特征范数。
- 在更宽步长范围内保持稳定，部分条件下可容忍任意大步长。
- 适合对超参数敏感的强化学习应用，尤其初学者或工程落地场景。
Q-learning和SARSA是强化学习的基础算法，其实际表现高度依赖步长设置。过大步长导致数值不稳定，过小则进展缓慢。本文提出隐式版本的Q-learning和SARSA，将迭代更新重构成不动点方程，实现步长自适应调整，其大小与特征范数成反比，无需人工调参即可提供自动正则化。非渐近分析表明，隐式方法在显著更广的步长范围内保持稳定；在有利条件下，允许任意大的步长并维持相近收敛速率。在涵盖离散与连续状态空间的基准环境上验证，隐式方法对步长选择的敏感性大幅降低，使用会使标准方法失效的步长仍能实现稳定性能。
原文摘要 · Abstract (English)
Q-learning and SARSA are foundational reinforcement learning algorithms whose practical success depends critically on step-size calibration. Step-sizes that are too large can cause numerical instability, while step-sizes that are too small can lead to slow progress. We propose implicit variants of Q-learning and SARSA that reformulate their iterative updates as fixed-point equations. This yields an adaptive step-size adjustment that scales inversely with feature norms, providing automatic regularization without manual tuning. Our non-asymptotic analyses demonstrate that implicit methods maintain stability over significantly broader step-size ranges. Under favorable conditions, it permits arbitrarily large step-sizes while achieving comparable convergence rates. Empirical validation across benchmark environments spanning discrete and continuous state spaces shows that implicit Q-learning and SARSA exhibit substantially reduced sensitivity to step-size selection, achieving stable performance with step-sizes that would cause standard methods to fail.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。