用评分数据训练强化学习奖励函数，效果优于传统偏好方法。
Reward Learning through Ranking Mean Squared Error
- 基于评分数据设计新的排序均方误差损失，直接学习轨迹评分
- 在机器人任务中表现稳定，优于现有评分与偏好学习方法
- 理论保证解集最小且完整，适合需要可靠奖励的场景
奖励设计仍是强化学习应用于现实问题的主要瓶颈。一种替代方案是奖励学习，即从人类反馈中推断奖励函数而非人工指定。近期工作提出从人类评分而非传统二元偏好中学习奖励函数，可提供更丰富且认知负担更低的监督信号。在此基础上，我们提出一种新的评分型强化学习方法R4。其核心是使用新型排序均方误差损失，从轨迹-评分对数据集中学习，将人类提供的离散评分（如差、中性、好）视为序数目标。与以往评分方法不同，R4在弱假设下具有形式化保证：其解集被证明是最小且完整的。实验表明，在使用人工和模拟评分数据时，R4在OpenAI Gym和DeepMind Control Suite的机器人基准上始终达到或超越现有评分与偏好学习方法。代码已开源：https://github.com/IRLL/R4。
原文摘要 · Abstract (English)
Reward design remains a significant bottleneck in applying reinforcement learning (RL) to real-world problems. A popular alternative is reward learning, where reward functions are inferred from human feedback rather than manually specified. Recent work has proposed learning reward functions from human ratings rather than traditional binary preferences, enabling richer and potentially less cognitively demanding supervision. Building on this paradigm, we introduce a new rating-based RL method, Ranked Return Regression for RL (R4). At its core, R4 uses a novel ranking mean squared error loss that learns from a dataset of trajectory-rating pairs, treating the human-provided discrete ratings (e.g., bad, neutral, good) as ordinal targets. Unlike prior rating-based approaches, R4 offers formal guarantees: its solution set is provably minimal and complete under mild assumptions. Empirically, using both human-provided and simulated ratings, we demonstrate that R4 consistently matches or outperforms existing rating and preference-based RL methods on robotic benchmarks from OpenAI Gym and the DeepMind Control Suite. Code released at https://github.com/IRLL/R4.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。