动态调整环境上下文，提升强化学习模型泛化能力
Dynamic Context Scheduling: Learning Beyond the Static Universe
- 用预设时序调度动态改变训练中的环境参数
- 在分布外场景下表现优于静态参数设置，复杂任务中也提升分布内性能
- 自动搜索多阶段调度策略，效果接近人工精心设计的方案
本文将动态上下文调度作为情境强化学习的训练工具。不同于将回合内上下文变化视为部署时的现实问题，我们将其视为可控的塑造机制：在每个训练回合中，根据预定调度让上下文动态演化，使策略接触更丰富且具有时间结构的环境参数空间。我们提出 DYNAMICCARLENV 框架，可接入多种可插拔调度类型，如正弦偏移或余弦退火。在加入 CARL 上下文化的 CartPole、BipedalWalker 与 VehicleRacing 环境中，动态调度在分布外（OOD）情形下表现匹配或超越静态基线；有趣的是，在更复杂的 BipedalWalker 与 VehicleRacing 中，其分布内（ID）评估性能也更高。初步结果表明，自动搜索多阶段课程能成功发现提升泛化性的调度策略，性能可媲美对单阶段调度进行广泛网格搜索的结果。
原文摘要 · Abstract (English)
We study dynamic context scheduling as a training instrument for contextual re- inforcement learning. Rather than treating intra-episode context variation as a deployment reality, we treat it as a controlled shaping mechanism. Thereby, context evolves within each training episode according to a predetermined schedule, expos- ing the policy to a richer and more temporally structured region of the environment parameter space. We introduce DYNAMICCARLENV, a framework that wraps contextual environments with pluggable schedule families, such as sinusoidal off- sets or cosine annealing. Across CartPole, BipedalWalker and VehicleRacing with CARL contextualization, we show that dynamic schedules match or outperform static context baselines in the out-of-distribution (OOD) regimes. Interestingly, for the more complex BipedalWalker and VehicleRacing environments we also achieve higher in-distribution (ID) evaluation performance. Preliminary findings indicate that automatic search for multi-stage curricula can successfully discover schedules that improve generalization, performing comparably to extensive grid search over single-stage schedulers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。