arXiv:2607.28916cs.LGcs.AI2026-07

提出一种新框架,让Q-learning在离线学习时能灵活控制偏差与信用分配长度的权衡。

Gated Q-learning: Add Off-Policy Bias to Taste

论文配图:Gated Q-learning: Add Off-Policy Bias to Taste
图 1 · 摘自论文原文
  • 用状态动作相关的门控机制替代重要性采样,动态调节信用分配范围。
  • 实验证明中间门控策略比传统方法更快收敛,且保持值估计稳定性。
  • 适合需要平衡学习速度与误差控制的强化学习应用,如机器人控制。

多步信用分配对样本高效的强化学习至关重要,但管理Q-learning中的离线偏差仍是一大挑战。三十年来,从业者只能在两个极端间二选一:消除偏差会严重截断资格迹(Watkins' Q(λ)),或忽略偏差虽加速学习却引入有害误差(Peng's Q(λ))。现代离线估计器未能解决此矛盾,因重要性采样比率在Q-learning的贪心目标策略下会崩溃。我们提出Gated Q-learning,一种新型算法框架,通过连续、状态-动作依赖的门控机制,有选择性地衰减资格迹,实现探索感知的调节。我们提供了严格的理论基础,证明期望算子仍是压缩映射,并推导出其精确不动点。实证表明,中间门控可安全延长信用分配范围,相比两端方法初始学习更快。该方法为重要性采样提供简单替代方案,同时允许定制有效多步范围和离线偏差量。

原文摘要 · Abstract (English)

Multistep credit assignment is critical for sample-efficient reinforcement learning, yet managing off-policy bias in Q-learning remains a fundamental challenge. For 30 years, practitioners have been limited to a binary choice: eliminate the bias at the cost of severely truncated eligibility traces (Watkins' Q($λ$)), or ignore the bias to learn faster while injecting detrimental errors into the value estimates (Peng's Q($λ$)). Modern off-policy estimators fail to resolve this tension, as importance-sampling ratios collapse under Q-learning's greedy target policy. We introduce Gated Q-learning, a novel algorithmic framework that ends this dilemma by smoothly interpolating between the two historical extremes. Rather than relying on importance sampling, our approach employs a continuous, state-action-dependent gating mechanism to selectively attenuate eligibility traces in an exploration-aware manner. We provide a rigorous theoretical foundation for this mechanism, proving that the expected operator remains a contraction mapping and deriving its exact fixed point. Empirical evaluations verify that intermediate gating safely enables longer credit-assignment horizons, yielding faster initial learning than either extreme. Gated Q-learning offers a simple alternative to importance sampling while enabling customization of the effective multistep horizon and the amount of off-policy bias in Q-learning agents.

强化学习信用分配离线学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。