提出可证明高效的偏好强化学习算法，兼顾风险感知与人类反馈对齐。
RA-PbRL: Provably Efficient Risk-Aware Preference-Based Reinforcement Learning
- 设计新算法RA-PbRL，支持嵌套与静态分位数风险目标
- 理论证明后悔上界为亚线性，随轮次增长缓慢上升
- 适合高风险场景如医疗、自动驾驶中的安全对齐任务
基于人类反馈的强化学习（RLHF）近年来备受关注，尤其在对齐大语言模型等人工智能系统与人类意图方面。本质上，RLHF是偏好强化学习（PbRL）的一种特殊形式，其偏好来源于人类判断而非任意评估者。尽管存在此关联，当前多数RLHF与PbRL方法仍聚焦于均值奖励优化，忽视了如人工智能安全、医疗和自动驾驶等需要风险感知的场景。这些场景通常采用单回合奖励设定，使传统风险敏感目标难以适用。为此，本文探讨并证明了两种风险感知目标在PbRL中的适用性：嵌套与静态分位数风险目标，并提出风险感知偏好强化学习（RA-PbRL）算法以优化这两类目标。同时，本文提供了理论分析，证明其后悔上界关于回合数为亚线性；并通过实证结果验证了理论发现。代码已开源。
原文摘要 · Abstract (English)
Reinforcement Learning from Human Feedback (RLHF) has recently surged in popularity, particularly for aligning large language models and other AI systems with human intentions. At its core, RLHF can be viewed as a specialized instance of Preference-based Reinforcement Learning (PbRL), where the preferences specifically originate from human judgments rather than arbitrary evaluators. Despite this connection, most existing approaches in both RLHF and PbRL primarily focus on optimizing a mean reward objective, neglecting scenarios that necessitate risk-awareness, such as AI safety, healthcare, and autonomous driving. These scenarios often operate under a one-episode-reward setting, which makes conventional risk-sensitive objectives inapplicable. To address this, we explore and prove the applicability of two risk-aware objectives to PbRL : nested and static quantile risk objectives. We also introduce Risk-AwarePbRL (RA-PbRL), an algorithm designed to optimize both nested and static objectives. Additionally, we provide a theoretical analysis of the regret upper bounds, demonstrating that they are sublinear with respect to the number of episodes, and present empirical results to support our findings. Our code is available in https://github.com/aguilarjose11/PbRLNeurips.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。