通过多策略对比提升人类偏好判断效率，实现更少查询的机器人技能学习
DAPPER: Discriminability-Aware Policy-to-Policy Preference-Based Reinforcement Learning for Query-Efficient Robot Skill Acquisition
- 用多个新训练策略生成对比轨迹，打破单策略导致的多样性不足
- 引入可学习的判别度指标，优先选择人类更容易区分的对比项
- 在仿真和真实四足机器人上显著降低所需查询次数，尤其在模糊偏好场景下
基于偏好的强化学习（PbRL）通过简单的人类轨迹对比即可实现策略学习。然而，传统方法因单一策略导致轨迹多样性差，使人类难以分辨优劣，造成查询效率低下。本文提出将‘偏好判别度’作为核心指标，衡量人类判断哪个轨迹更接近理想行为的难易程度。为提升判别度，DAPPER不局限于单策略内对比，而是通过从零训练多个策略生成多样化轨迹。该方法在每次奖励更新后重训练新策略，并引入判别度判别器，自动估计每对轨迹的可区分性，从而优先采样更易判断的对比样本。训练中联合优化偏好奖励与判别度得分，推动发现既高分又易区分的策略。在模拟与真实四足机器人环境中验证表明，DAPPER显著优于现有方法，在低判别度等挑战条件下仍保持高效，大幅减少所需人类查询次数。
原文摘要 · Abstract (English)
Preference-based Reinforcement Learning (PbRL) enables policy learning through simple queries comparing trajectories from a single policy. While human responses to these queries make it possible to learn policies aligned with human preferences, PbRL suffers from low query efficiency, as policy bias limits trajectory diversity and reduces the number of discriminable queries available for learning preferences. This paper identifies preference discriminability, which quantifies how easily a human can judge which trajectory is closer to their ideal behavior, as a key metric for improving query efficiency. To address this, we move beyond comparisons within a single policy and instead generate queries by comparing trajectories from multiple policies, as training them from scratch promotes diversity without policy bias. We propose Discriminability-Aware Policy-to-Policy Preference-Based Efficient Reinforcement Learning (DAPPER), which integrates preference discriminability with trajectory diversification achieved by multiple policies. DAPPER trains new policies from scratch after each reward update and employs a discriminator that learns to estimate preference discriminability, enabling the prioritized sampling of more discriminable queries. During training, it jointly maximizes the preference reward and preference discriminability score, encouraging the discovery of highly rewarding and easily distinguishable policies. Experiments in simulated and real-world legged robot environments demonstrate that DAPPER outperforms previous methods in query efficiency, particularly under challenging preference discriminability conditions. A supplementary video that facilitates understanding of the proposed framework and its experimental results is available at: https://youtu.be/lRwX8FNN8n4
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。