用学生能力反推环境，减少资源受限下的训练交互次数。
Efficient Unsupervised Environment Design through Hierarchical Policy Representation Learning
- 构建分层MDP框架，让教师根据学生策略表示生成适配环境。
- 实验表明在单轮中交互次数更少，性能仍优于基线方法。
- 适合计算资源有限或交互成本高的智能体训练场景。
无监督环境设计（UED）通过自动化课程生成，为开发通用智能体提供了新路径。现有方法依赖随机过程实现开放性生成，但在资源受限场景下不切实际。本文提出一种分层马尔可夫决策过程（MDP）框架，教师智能体基于从发现的评估环境中提取的学生策略表示，生成适配其能力的训练环境。为提升效率，引入生成模型，用合成数据扩充教师训练集，降低师生交互需求。在多个领域实验中，本方法在单次回合内所需交互次数更少，且性能优于基线。结果表明该方法适用于训练机会有限的场景。
原文摘要 · Abstract (English)
Unsupervised Environment Design (UED) has emerged as a promising approach to developing general-purpose agents through automated curriculum generation. Popular UED methods focus on Open-Endedness, where teacher algorithms rely on stochastic processes for infinite generation of useful environments. This assumption becomes impractical in resource-constrained scenarios where teacher-student interaction opportunities are limited. To address this challenge, we introduce a hierarchical Markov Decision Process (MDP) framework for environment design. Our framework features a teacher agent that leverages student policy representations derived from discovered evaluation environments, enabling it to generate training environments based on the student's capabilities. To improve efficiency, we incorporate a generative model that augments the teacher's training dataset with synthetic data, reducing the need for teacher-student interactions. In experiments across several domains, we show that our method outperforms baseline approaches while requiring fewer teacher-student interactions in a single episode. The results suggest the applicability of our approach in settings where training opportunities are limited.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。