通过改进后悔值估计与任务关联建模，提升强化学习环境设计的泛化能力。
TRACED: Transition-aware Regret Approximation with Co-learnability for Environment Design
- 引入状态转移预测误差和共学习性度量，优化环境生成的后悔值估计
- 在多个基准上实现比强基线更好的零样本泛化性能
- 适合研究自适应课程设计与强化学习泛化的研究人员
将深度强化学习代理推广到未见环境仍面临重大挑战。一种有前景的解决方案是无监督环境设计（UED），即教师动态生成具有高学习潜力的任务，学生从不断演化的课程中学习鲁棒策略。现有UED方法通常通过后悔值衡量学习潜力，即最优与当前表现的差距，该值仅由价值函数损失近似。本文在此基础上，引入状态转移预测误差作为后悔值估计的额外项，并提出轻量级度量“共学习性”以捕捉训练一个任务对其他任务性能的影响。结合这两项，提出过渡感知后悔值近似与共学习性环境设计方法（TRACED）。实验表明，TRACED生成的课程在多个基准上显著优于强基线，实现更优的零样本泛化。消融实验证明，状态转移预测误差驱动复杂度快速提升，而共学习性在与该误差结合时带来额外增益。结果表明，精细的后悔值估计与显式任务关系建模可有效支持样本高效的无监督课程设计。
原文摘要 · Abstract (English)
Generalizing deep reinforcement learning agents to unseen environments remains a significant challenge. One promising solution is Unsupervised Environment Design (UED), a co-evolutionary framework in which a teacher adaptively generates tasks with high learning potential, while a student learns a robust policy from this evolving curriculum. Existing UED methods typically measure learning potential via regret, the gap between optimal and current performance, approximated solely by value-function loss. Building on these approaches, we introduce the transition-prediction error as an additional term in our regret approximation. To capture how training on one task affects performance on others, we further propose a lightweight metric called Co-Learnability. By combining these two measures, we present Transition-aware Regret Approximation with Co-learnability for Environment Design (TRACED). Empirical evaluations show that TRACED produces curricula that improve zero-shot generalization over strong baselines across multiple benchmarks. Ablation studies confirm that the transition-prediction error drives rapid complexity ramp-up and that Co-Learnability delivers additional gains when paired with the transition-prediction error. These results demonstrate how refined regret approximation and explicit modeling of task relationships can be leveraged for sample-efficient curriculum design in UED. Project Page: https://geonwoo.me/traced/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。