提出新算法实现离线强化学习零样本泛化，让模型在未见环境表现良好。
Provable Zero-Shot Generalization in Offline Reinforcement Learning
- 用悲观评估引导策略学习，提升对未知环境的适应能力。
- 理论证明两种算法可找到接近最优的泛化策略。
- 适合研究离线强化学习泛化机制的学者参考。
本文研究具有零样本泛化能力（ZSG）的离线强化学习（Offline RL），即智能体仅通过包含不同环境经验的离线数据集进行训练，目标是使策略在未见过的测试环境中无需进一步交互即可表现良好。已有工作表明经典离线RL无法泛化至新环境。本文提出悲观经验风险最小化（PERM）和悲观近端策略优化（PPPO），利用悲观策略评估指导策略学习以增强泛化性能。理论分析表明，两者均能发现具备零样本泛化能力的近优策略。该结果为理解离线强化学习泛化现象提供了首个理论基础。
原文摘要 · Abstract (English)
In this work, we study offline reinforcement learning (RL) with zero-shot generalization property (ZSG), where the agent has access to an offline dataset including experiences from different environments, and the goal of the agent is to train a policy over the training environments which performs well on test environments without further interaction. Existing work showed that classical offline RL fails to generalize to new, unseen environments. We propose pessimistic empirical risk minimization (PERM) and pessimistic proximal policy optimization (PPPO), which leverage pessimistic policy evaluation to guide policy learning and enhance generalization. We show that both PERM and PPPO are capable of finding a near-optimal policy with ZSG. Our result serves as a first step in understanding the foundation of the generalization phenomenon in offline reinforcement learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。