用无奖励表征提升离线偏好强化学习效率
From Reward-Free Representations to Preferences: Rethinking Offline Preference-Based Reinforcement Learning
- 先学无奖励表征，再用偏好数据对比优化
- 在相同偏好样本下，性能优于现有方法
- 适合偏好数据稀缺的场景，如人机交互
基于偏好的强化学习（PbRL）通过成对的人类偏好反馈避免显式奖励设计。现有离线PbRL方法通常采用两阶段流程：先从标注偏好中学习奖励或偏好模型，再在未标注数据上进行离线强化学习。本文从零样本强化学习中的无奖励表征学习（RFRL）视角重新审视离线PbRL，提出新训练框架：首先从无奖励离线数据中学习潜在的后继度量表征，然后利用偏好数据进行对比搜索与微调。大量实验与消融分析表明，该方法在偏好效率上显著优于现有离线PbRL基线。这是首个将RFRL与PbRL结合的工作，揭示其作为高效反馈解决方案的潜力。代码已公开于https://github.com/rl-bandits-lab/FB-PbRL。
原文摘要 · Abstract (English)
Preference-based reinforcement learning (PbRL) avoids explicit reward engineering by learning from pairwise human preference feedback. Existing offline PbRL methods typically follow a two-stage pipeline, first learning a reward or preference model from labeled preferences and then performing offline RL on unlabeled data. We revisit offline PbRL through the lens of reward-free representation learning (RFRL) from the zero-shot RL literature, and propose a new training framework that first learns latent successor-measure representations from reward-free offline data, followed by contrastive search and fine-tuning using preference data. Through extensive experiments and ablations, we show that our method achieves superior preference efficiency over offline PbRL baselines. This work is the first to connect RFRL with PbRL, highlighting its potential as a feedback-efficient solution. Our code is publicly available at https://github.com/rl-bandits-lab/FB-PbRL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。