让模型同时学习多种人类评分策略，更真实地模仿人的决策方式。
Multi-Task Reward Learning from Human Ratings
- 通过联合建模分类与回归任务，动态学习人类评分权重。
- 在合成评分数据上表现优于现有基于评分的强化学习方法。
- 适合需要多维度用户反馈对齐的任务，如对话系统优化。
从人类反馈中进行强化学习（RLHF）已成为使模型行为与用户目标对齐的关键。然而，当前方法常将人类决策简化为单一任务（如分类或回归），而人类实际决策涉及多种策略的综合。本文提出一种新强化学习方法，通过联合考虑多个任务来模拟人类决策过程。具体而言，利用无奖励环境中的用户评分推断奖励函数，并引入可学习权重以平衡分类与回归模型的贡献。该设计捕捉了人类判断中的固有不确定性，使模型能自适应强调不同策略。我们在合成人类评分数据上进行了多组实验，结果表明，该方法在多数情况下优于现有基于评分的强化学习方法，部分场景甚至超越传统强化学习方法。
原文摘要 · Abstract (English)
Reinforcement learning from human feedback (RLHF) has become a key factor in aligning model behavior with users' goals. However, while humans integrate multiple strategies when making decisions, current RLHF approaches often simplify this process by modeling human reasoning through isolated tasks such as classification or regression. In this paper, we propose a novel reinforcement learning (RL) method that mimics human decision-making by jointly considering multiple tasks. Specifically, we leverage human ratings in reward-free environments to infer a reward function, introducing learnable weights that balance the contributions of both classification and regression models. This design captures the inherent uncertainty in human decision-making and allows the model to adaptively emphasize different strategies. We conduct several experiments using synthetic human ratings to validate the effectiveness of the proposed approach. Results show that our method consistently outperforms existing rating-based RL methods, and in some cases, even surpasses traditional RL approaches.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。