用后悔值引导扩散模型生成挑战性环境，提升智能体泛化能力
Adversarial Environment Design via Regret-Guided Diffusion Models
- 用智能体的后悔值指导扩散模型生成环境
- 在新环境中零样本泛化性能超越基线方法
- 适合需要鲁棒训练的强化学习场景
在深度强化学习中，使智能体具备应对环境变化的鲁棒性仍是重大挑战。无监督环境设计（UED）通过生成与智能体能力匹配的训练环境来应对该问题。然而，现有方法受限于环境生成能力。为此，本文提出一种新型UED算法——基于后悔值引导扩散模型的对抗环境设计（ADD）。该方法利用智能体的后悔值引导基于扩散模型的环境生成器，生成既具挑战性又利于智能体改进的环境。借助扩散模型的强大表征能力，ADD可直接生成对抗性环境，同时保持训练环境多样性，从而帮助智能体有效学习鲁棒策略。实验表明，该方法成功生成了具有指导性的环境课程，在新颖的、分布外的环境中实现了优于现有基线的零样本泛化性能。
原文摘要 · Abstract (English)
Training agents that are robust to environmental changes remains a significant challenge in deep reinforcement learning (RL). Unsupervised environment design (UED) has recently emerged to address this issue by generating a set of training environments tailored to the agent's capabilities. While prior works demonstrate that UED has the potential to learn a robust policy, their performance is constrained by the capabilities of the environment generation. To this end, we propose a novel UED algorithm, adversarial environment design via regret-guided diffusion models (ADD). The proposed method guides the diffusion-based environment generator with the regret of the agent to produce environments that the agent finds challenging but conducive to further improvement. By exploiting the representation power of diffusion models, ADD can directly generate adversarial environments while maintaining the diversity of training environments, enabling the agent to effectively learn a robust policy. Our experimental results demonstrate that the proposed method successfully generates an instructive curriculum of environments, outperforming UED baselines in zero-shot generalization across novel, out-of-distribution environments. Project page: https://rllab-snu.github.io/projects/ADD
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。