让智能体学会避开不良示范，而非模仿专家行为。
UNIQ: Offline Inverse Q-learning for Avoiding Undesirable Demonstrations
- 通过最大化策略与不良行为在状态动作分布上的统计距离，实现避障学习。
- 在标准环境上表现优于现有最优基线，有效避免了不良示范的影响。
- 基于逆Q-learning框架，可高效利用无标签数据，适合安全关键场景。
我们研究离线学习策略以规避不良示范的问题。与传统离线模仿学习旨在模仿专家或近似最优示范不同，本文关注的是避免不良行为（由不良示范指定）。为此，我们不再最小化学习策略与专家示范之间的距离，而是将其建模为最大化学习策略与不良策略在状态-动作平稳分布空间中的统计距离。这一新思路催生了独特的训练目标，并要求开发新算法。我们的算法UNIQ基于逆Q-learning框架，将学习问题视为非对抗性的合作任务，并展示了如何高效利用无标签数据进行实际训练。在标准基准环境上的实验表明，该方法始终优于当前最先进的基线。代码可在 https://github.com/hmhuy0/UNIQ 获取。
原文摘要 · Abstract (English)
We address the problem of offline learning a policy that avoids undesirable demonstrations. Unlike conventional offline imitation learning approaches that aim to imitate expert or near-optimal demonstrations, our setting involves avoiding undesirable behavior (specified using undesirable demonstrations). To tackle this problem, unlike standard imitation learning where the aim is to minimize the distance between learning policy and expert demonstrations, we formulate the learning task as maximizing a statistical distance, in the space of state-action stationary distributions, between the learning policy and the undesirable policy. This significantly different approach results in a novel training objective that necessitates a new algorithm to address it. Our algorithm, UNIQ, tackles these challenges by building on the inverse Q-learning framework, framing the learning problem as a cooperative (non-adversarial) task. We then demonstrate how to efficiently leverage unlabeled data for practical training. Our method is evaluated on standard benchmark environments, where it consistently outperforms state-of-the-art baselines. The code implementation can be accessed at: https://github.com/hmhuy0/UNIQ.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。