用专家偏好估算奖励，让强化学习在长任务中更稳定高效。
ELO-Rated Sequence Rewards: Advancing Reinforcement Learning Models
- 基于专家对轨迹的排序偏好，用ELO评分生成奖励信号。
- 在5000步的长期任务中超越多个主流基线方法。
- 适合缺乏明确奖励函数的复杂长周期强化学习场景。
强化学习（RL）严重依赖精心设计的奖励函数。然而，在长期强化学习（LTRL）任务中，准确为每个状态-动作对分配奖励仍是重大挑战，导致代理通常需专家指导。受经济学中的序数效用理论启发，我们提出一种新型奖励估计算法：基于ELO评分的强化学习（ERRL）。该方法有两个关键贡献：首先，利用专家对轨迹的偏好而非数值奖励（效用）来计算每条轨迹的ELO评分作为其奖励；其次，引入新的奖励重分配算法，缓解无固定锚定奖励时的训练不稳定性。在长达5000步的长期场景中，传统RL算法表现不佳，而我们的方法显著优于多个先进基线。此外，我们还全面分析了专家偏好对结果的影响。
原文摘要 · Abstract (English)
Reinforcement Learning (RL) heavily relies on the careful design of the reward function. However, accurately assigning rewards to each state-action pair in Long-Term Reinforcement Learning (LTRL) tasks remains a significant challenge. As a result, RL agents are often trained under expert guidance. Inspired by the ordinal utility theory in economics, we propose a novel reward estimation algorithm: ELO-Rating based Reinforcement Learning (ERRL). This approach features two key contributions. First, it uses expert preferences over trajectories rather than cardinal rewards (utilities) to compute the ELO rating of each trajectory as its reward. Second, a new reward redistribution algorithm is introduced to alleviate training instability in the absence of a fixed anchor reward. In long-term scenarios (up to 5000 steps), where traditional RL algorithms struggle, our method outperforms several state-of-the-art baselines. Additionally, we conduct a comprehensive analysis of how expert preferences influence the results.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。