融合奖励与偏好信号，提升强化学习训练稳定性和性能
Fusing Rewards and Preferences in Reinforcement Learning
- 直接用策略日志概率建模偏好，无需额外奖励建模
- 在6个控制环境中表现优于或持平SAC，训练更稳定
- 适合有偏好数据但缺乏完整奖励函数的场景
我们提出Dual-Feedback Actor（DFA），一种将个体奖励与成对偏好（若可用）融合到单一更新规则中的强化学习算法。DFA直接利用策略的对数概率来建模偏好概率，避免了单独的奖励建模步骤。偏好可由人工标注（状态级或轨迹级）提供，也可在线从离线策略回放缓冲区存储的Q值中合成。在Bradley-Terry模型下，我们证明最小化DFA的偏好损失可恢复熵正则化的Soft Actor-Critic（SAC）策略。仿真结果表明，仅使用生成偏好训练的DFA在六个控制环境中的表现匹配或超越SAC，且训练过程更稳定。在随机网格世界中，仅需半合成偏好数据集，其性能即超过基于人类反馈的强化学习（RLHF）基线方法，并接近拥有真实奖励的最优解。
原文摘要 · Abstract (English)
We present Dual-Feedback Actor (DFA), a reinforcement learning algorithm that fuses both individual rewards and pairwise preferences (if available) into a single update rule. DFA uses the policy's log-probabilities directly to model the preference probability, avoiding a separate reward-modeling step. Preferences can be provided by human-annotators (at state-level or trajectory-level) or be synthesized online from Q-values stored in an off-policy replay buffer. Under a Bradley-Terry model, we prove that minimizing DFA's preference loss recovers the entropy-regularized Soft Actor-Critic (SAC) policy. Our simulation results show that DFA trained on generated preferences matches or exceeds SAC on six control environments and demonstrates a more stable training process. With only a semi-synthetic preference dataset under Bradley-Terry model, our algorithm outperforms reward-modeling reinforcement learning from human feedback (RLHF) baselines in a stochastic GridWorld and approaches the performance of an oracle with true rewards.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。