提出新方法量化环境新颖性,让无人监督训练更有效。
Improving Environment Novelty Quantification for Effective Unsupervised Environment Design
- 用学生状态动作空间覆盖度衡量环境新颖性
- 结合新颖性和遗憾值,提升训练效果
- 适用于强化学习中的自适应环境设计
无监督环境设计(UED)通过教师与学生智能体的交互,形成自适应课程以提升智能体处理未知场景的能力。现有方法主要依赖‘遗憾’指标来引导课程生成,但忽视了环境新颖性这一关键因素。由于环境参数定义模糊,新颖性测量极具挑战。本文提出覆盖度驱动的新颖性评估框架(CENIE),利用学生在历史课程中的状态-动作空间覆盖情况,实现可扩展、领域无关且课程感知的新颖性量化。具体实现中采用高斯混合模型建模覆盖分布并计算新颖性。将新颖性与遗憾值联合优化,使课程在逐步增加复杂度的同时有效探索状态-动作空间。实证表明,将CENIE集成至现有基于遗憾的UED算法,在多个基准上达到当前最优性能,验证了新颖性驱动自适应课程对鲁棒泛化的重要性。
原文摘要 · Abstract (English)
Unsupervised Environment Design (UED) formalizes the problem of autocurricula through interactive training between a teacher agent and a student agent. The teacher generates new training environments with high learning potential, curating an adaptive curriculum that strengthens the student's ability to handle unseen scenarios. Existing UED methods mainly rely on regret, a metric that measures the difference between the agent's optimal and actual performance, to guide curriculum design. Regret-driven methods generate curricula that progressively increase environment complexity for the student but overlook environment novelty -- a critical element for enhancing an agent's generalizability. Measuring environment novelty is especially challenging due to the underspecified nature of environment parameters in UED, and existing approaches face significant limitations. To address this, this paper introduces the Coverage-based Evaluation of Novelty In Environment (CENIE) framework. CENIE proposes a scalable, domain-agnostic, and curriculum-aware approach to quantifying environment novelty by leveraging the student's state-action space coverage from previous curriculum experiences. We then propose an implementation of CENIE that models this coverage and measures environment novelty using Gaussian Mixture Models. By integrating both regret and novelty as complementary objectives for curriculum design, CENIE facilitates effective exploration across the state-action space while progressively increasing curriculum complexity. Empirical evaluations demonstrate that augmenting existing regret-based UED algorithms with CENIE achieves state-of-the-art performance across multiple benchmarks, underscoring the effectiveness of novelty-driven autocurricula for robust generalization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。