用非对称损失改进多步Q学习，减少长时序偏差，提升训练效率。
Upper-Expectile Multi-Step Q-Learning for Off-Policy Reinforcement Learning
- 引入期望分位数损失替代传统TD损失，仅需调节一个超参数τ。
- 在确定性环境下τ=1时无偏差，且满足长时序学习的下界约束。
- 实测在27个任务中性能媲美先进方法，更适配多评论家扩展。
多步回报可加速离策略强化学习中的奖励传播，但会将每个决策的评估与后续次优动作耦合，导致随步长增长的悲观偏差。本文提出期望分位数n步Q学习（ENQ），将对称n步TD损失替换为针对动作价值误差的非对称期望分位数损失，仅新增一个期望分位数水平τ作为方法特有超参数。证明了ENQ算子是γⁿ-收缩的。在确定性动态下，当τ=1时，其在覆盖的支撑点对上于最优动作价值函数Q*处偏差消失，对应的不动点满足长时序Q学习（LQL）所用的分离-n实例及其倍数的下界不等式。在随机动态下，算子偏差具有与步长无关的双侧边界，噪声常数恒定。使用单一期望分位数水平τ=0.8和固定备份步长，在27个操作与导航任务实例上，ENQ在总体表现上与LQL相当，训练步数吞吐量更高，且在控制扩展实验中更受益于十评论家集成。
原文摘要 · Abstract (English)
Multi-step returns accelerate reward propagation in off-policy reinforcement learning, but couple the evaluation of each decision to the suboptimal logged actions that follow it, inducing a pessimistic bias that grows with the horizon. We propose Expectile $n$-step Q-learning (ENQ), which replaces the symmetric $n$-step temporal-difference (TD) loss with an asymmetric expectile loss on the action-value error, with expectile level $τ$ as the only method-specific hyperparameter added beyond $n$-step TD. We prove that the ENQ operator is a $γ^{n}$-contraction. Under deterministic dynamics, at $τ=1$, its bias vanishes at the optimal action-value function $Q^*$ on covered in-support pairs, and the corresponding fixed point satisfies the separation-$n$ instance and its multiples of the lower-bound inequality used by Long-Horizon Q-learning (LQL). Under stochastic dynamics, the operator bias admits two-sided bounds with horizon-independent noise constants. Using a single expectile level $τ=0.8$ and a fixed backup horizon across 27 manipulation and navigation task instances, ENQ is competitive with LQL on aggregate, achieves higher measured training-step throughput in our profiling study, and benefits more from a ten-critic ensemble in a controlled scaling experiment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。