用语言理由提升奖励学习的因果鲁棒性，避免学错无关特征。
Causally Robust Reward Learning from Reason-Augmented Preference Feedback
- 通过语言理由构建嵌入空间中的引导轴，聚焦相关特征
- 分布外场景下奖励准确率提升1.5倍，新任务策略性能提升2倍
- 无需额外数据或微调，跨任务复用因果方向，适合需可解释性的场景
基于偏好反馈的奖励学习广泛用于对齐智能体行为与用户偏好，但稀疏的二值反馈易引发因果混淆。模型常误将训练中与偏好轨迹共现的虚假特征当作关键依据，导致测试时因相关性消失或反转而失效。我们提出ReCouPLe——一个轻量级框架，利用自然语言理由提供缺失的因果信号。每个理由被视为嵌入空间中的引导投影轴，训练模型根据与该轴对齐的特征评分轨迹，同时弱化与理由无关的上下文。由于相同理由（如“避免碰撞”、“更快完成任务”）可在多任务中重复出现，ReCouPLe能自然复用相同的因果方向，实现偏好知识向新任务迁移，且无需额外数据或语言模型微调。所学奖励模型能基于明确理由锚定偏好，更贴近用户意图，并超越虚假特征泛化。在分布外设置下，奖励准确率最高提升1.5倍；在新任务上，下游策略性能提升2倍。代码已开源：https://github.com/mj-hwang/ReCouPLe。
原文摘要 · Abstract (English)
Preference-based reward learning is widely used for shaping agent behavior to match a user's preference, yet its sparse binary feedback makes it especially vulnerable to causal confusion. The learned reward often latches onto spurious features that merely co-occur with preferred trajectories during training, collapsing when those correlations disappear or reverse at test time. We introduce ReCouPLe, a lightweight framework that uses natural language rationales to provide the missing causal signal. Each rationale is treated as a guiding projection axis in an embedding space, training the model to score trajectories based on features aligned with that axis while de-emphasizing context that is unrelated to the stated reason. Because the same rationales (e.g., "avoids collisions", "completes the task faster") can appear across multiple tasks, ReCouPLe naturally reuses the same causal direction whenever tasks share semantics, and transfers preference knowledge to novel tasks without extra data or language-model fine-tuning. Our learned reward model can ground preferences on the articulated reason, aligning better with user intent and generalizing beyond spurious features. ReCouPLe outperforms baselines by up to 1.5x in reward accuracy under distribution shifts, and 2x in downstream policy performance in novel tasks. We have released our code at https://github.com/mj-hwang/ReCouPLe
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。