考虑人类特质提升强化学习中人类反馈建模精度
CHARM: Considering Human Attributes for Reinforcement Modeling
- 引入人类特质（如机器人经验、教育背景）改进反馈建模
- 实验证明加入人类特质可更准确预测反馈价值
- 适用于需要高质量人类反馈的AI训练场景
基于人类反馈的强化学习在多个领域取得显著进展，其性能高度依赖于反馈质量。尽管已有研究指出人类教师特征会影响反馈模式，但缺乏对实际影响的深入探究。本文设计了一项探索性研究，考察人类反馈模式与人类特征之间的关联。通过在公共空间开展两项长周期任务实验，共招募46名参与者。结果表明，反馈模式不仅与任务统计量（如奖励值）相关，还与参与者特征密切相关，尤其是机器人经验和教育背景。此外，相比仅使用任务统计量，结合人类特征可更准确预测人类反馈价值。所有收集的人类反馈数据与特征，以及数据采集和反馈预测代码均已公开在https://github.com/AABL-Lab/CHARM。
原文摘要 · Abstract (English)
Reinforcement Learning from Human Feedback has recently achieved significant success in various fields, and its performance is highly related to feedback quality. While much prior work acknowledged that human teachers' characteristics would affect human feedback patterns, there is little work that has closely investigated the actual effects. In this work, we designed an exploratory study investigating how human feedback patterns are associated with human characteristics. We conducted a public space study with two long horizon tasks and 46 participants. We found that feedback patterns are not only correlated with task statistics, such as rewards, but also correlated with participants' characteristics, especially robot experience and educational background. Additionally, we demonstrated that human feedback value can be more accurately predicted with human characteristics compared to only using task statistics. All human feedback and characteristics we collected, and codes for our data collection and predicting more accurate human feedback are available at https://github.com/AABL-Lab/CHARM
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。