解决多阶段任务中反馈错位问题，提升强化学习对齐人类意图的能力。
STAIR: Addressing Stage Misalignment through Temporal-Aligned Preference Reinforcement Learning
- 基于时间距离的对比学习，自动划分任务阶段
- 优先比较同阶段片段，避免跨阶段无效反馈
- 与人类认知一致，适合复杂分步任务优化
基于偏好的强化学习（PbRL）通过直接从人类偏好中学习奖励，避免了复杂的奖励设计，更贴近人类意图。但在多阶段任务中，由于阶段错位——例如将移动与抓取阶段的片段进行比较——导致反馈信息不足，限制了策略学习效果。本文通过理论分析和实验证实了该问题的存在。为此提出STage-AlIgned Reward learning（STAIR），首先利用对比学习基于时间距离学习阶段近似，无需预设任务知识，并能动态适应策略变化；随后在相同阶段内优先进行偏好比较。大量实验表明，STAIR在多阶段任务中表现更优，在单阶段任务中也具备竞争力。此外，人类评估显示，STAIR识别出的阶段与人类认知高度一致，验证了其缓解阶段错位的有效性。
原文摘要 · Abstract (English)
Preference-based reinforcement learning (PbRL) bypasses complex reward engineering by learning rewards directly from human preferences, enabling better alignment with human intentions. However, its effectiveness in multi-stage tasks, where agents sequentially perform sub-tasks (e.g., navigation, grasping), is limited by stage misalignment: Comparing segments from mismatched stages, such as movement versus manipulation, results in uninformative feedback, thus hindering policy learning. In this paper, we validate the stage misalignment issue through theoretical analysis and empirical experiments. To address this issue, we propose STage-AlIgned Reward learning (STAIR), which first learns a stage approximation based on temporal distance, then prioritizes comparisons within the same stage. Temporal distance is learned via contrastive learning, which groups temporally close states into coherent stages, without predefined task knowledge, and adapts dynamically to policy changes. Extensive experiments demonstrate STAIR's superiority in multi-stage tasks and competitive performance in single-stage tasks. Furthermore, human studies show that stages approximated by STAIR are consistent with human cognition, confirming its effectiveness in mitigating stage misalignment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。