用参数变化评估环境,让强化学习训练更高效。
PACE: Parameter Change for Unsupervised Environment Design

- 通过策略参数更新幅度衡量环境价值
- 在MiniGrid上实现96.4%的IQM和17.2%最优差距
- 无需额外采样,计算高效且方差低
无监督环境设计(UED)通过自适应调整训练环境来提升强化学习泛化能力,但其有效性依赖可靠的环境评估。现有方法多采用间接代理信号(如后悔值、基于值的误差或蒙特卡洛估计),存在偏差、高方差或计算开销大等问题,无法真实反映智能体的学习进展。为此,本文提出参数变化环境设计(PACE),通过训练过程中产生的策略参数变化来直接评估环境价值,将环境选择与实际学习进展对齐。具体而言,PACE使用策略优化目标的一阶近似,环境价值与对应参数更新的平方L2范数成正比,实现低方差、高效率的评估,无需额外轨迹采样。在MiniGrid和Craftax上的实验表明,PACE持续优于现有基准,尤其在分布外(OOD)评估中表现突出:在MiniGrid上达到96.4%的IQM和17.2%的最优差距。
原文摘要 · Abstract (English)
Unsupervised Environment Design (UED) offers a promising paradigm for improving reinforcement learning generalization by adaptively shaping training environments, but it requires reliable environment evaluation to remain effective. However, existing UED methods evaluate environments using indirect proxy signals such as regret, value-based errors, or Monte Carlo, which suffer from bias, high variance, or substantial computational overhead and fail to reflect agent realized learning progress. To address these limitations, we propose Parameter Change Environment Design (PACE), which evaluates an environment through the policy parameter change induced by training on that environment, directly grounding environment selection in realized learning progress. Specifically, PACE assigns environment value using a first-order approximation of the policy optimization objective, where the improvement induced by an environment is proportional to the squared L2 norm of the corresponding parameter update, enabling low-variance and computation-efficient evaluation without additional rollouts. Experiments on MiniGrid and Craftax show that PACE consistently outperforms established UED baselines, achieving higher IQM and smaller Optimality Gap on OOD evaluations, including an IQM of 96.4% and an Optimality Gap of 17.2% on MiniGrid.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。