arXiv:2510.15382cs.LGcs.AI2025-10NeurIPS被引 7

提升零样本强化学习的稳定性与泛化能力,解决旧方法在分布外动作下的偏差问题。

Towards Robust Zero-Shot Reinforcement Learning

  • 引入行为正则化,将策略优化转为稳定的学习范式
  • 用任务条件扩散模型生成高质量多模态动作分布
  • 采用注意力架构增强表征表达力,适合复杂环境任务

零样本强化学习(Zero-Shot RL)为预训练通用策略提供了新路径,使其能在零样本条件下适应任意新任务。尽管流行的前向-后向表示(FB)及其相关方法已展现出潜力,但我们发现其建模表达能力不足,且离线学习中分布外(OOD)动作导致的外推误差常引发表征偏差,最终影响性能。为此,我们提出行为正则化增强表达的零样本强化学习框架(BREEZE),该框架在基于FB的基础上,同时提升学习稳定性、策略提取能力和表征质量。BREEZE在零样本策略学习中引入行为正则化,将策略优化转化为稳定的分布内学习范式;通过任务条件扩散模型提取策略,实现零样本场景下高质量、多模态动作分布生成;并采用表达力强的注意力架构建模环境动态间的复杂关系。在ExORL和D4RL Kitchen上的大量实验表明,BREEZE达到最佳或接近最佳性能,且相比先前离线零样本强化学习方法具有更强鲁棒性。官方代码已开源:https://github.com/Whiterrrrr/BREEZE。

原文摘要 · Abstract (English)

The recent development of zero-shot reinforcement learning (RL) has opened a new avenue for learning pre-trained generalist policies that can adapt to arbitrary new tasks in a zero-shot manner. While the popular Forward-Backward representations (FB) and related methods have shown promise in zero-shot RL, we empirically found that their modeling lacks expressivity and that extrapolation errors caused by out-of-distribution (OOD) actions during offline learning sometimes lead to biased representations, ultimately resulting in suboptimal performance. To address these issues, we propose Behavior-REgularizEd Zero-shot RL with Expressivity enhancement (BREEZE), an upgraded FB-based framework that simultaneously enhances learning stability, policy extraction capability, and representation learning quality. BREEZE introduces behavioral regularization in zero-shot RL policy learning, transforming policy optimization into a stable in-sample learning paradigm. Additionally, BREEZE extracts the policy using a task-conditioned diffusion model, enabling the generation of high-quality and multimodal action distributions in zero-shot RL settings. Moreover, BREEZE employs expressive attention-based architectures for representation modeling to capture the complex relationships between environmental dynamics. Extensive experiments on ExORL and D4RL Kitchen demonstrate that BREEZE achieves the best or near-the-best performance while exhibiting superior robustness compared to prior offline zero-shot RL methods. The official implementation is available at: https://github.com/Whiterrrrr/BREEZE.

强化学习零样本扩散模型策略生成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。