用可观测性分数选模型检查点，提升强化学习在月球着陆任务中的性能。
Predicting Closed-Loop Performance of Latent World Models: Offline Checkpoint Selection for MPC and Model-Based RL Under Non-Markovian Rewards in LunarLander
- 从最优控制理论出发设计结构化验证指标
- 奖励可观察性分数是预测闭环性能最强的单一指标
- 适合做基于模型强化学习的工程师和研究者
我们研究如何仅通过验证阶段的诊断来预测学习到的隐变量世界模型的下游闭环性能。在世界模型训练过程中选择合适的检查点极具挑战：验证损失和多步预测均方根误差在闭环性能崩溃后仍持续改善。本文基于最优控制理论提出一套结构化验证诊断方法，并应用于具有奖励塑形的Gymnasium LunarLander v3环境。在该环境中训练一个RSSM[5,4]世界模型，将每个检查点的CEM-MPC回报作为闭环质量的基准。通过对40个指标与该基准进行对比，发现最强的单一预测因子是奖励可观察性分数（ROF），它衡量奖励预测器对可观测子空间的依赖程度。我们将ROF与三个结构正则项结合，形成单数值离线检查点选择评分——复合奖励可观察性分数（CROF）。使用CROF选出的世界模型训练出的基于模型A2C策略，在公平评估下比模型无关的A2C基线高出约24.5回报点，且仅需约65倍更少的真实环境交互次数；同一模型还可驱动表现优异的零样本CEM-MPC策略。代码与数据：https://github.com/nsmoly/LunarLander_RSSM。
原文摘要 · Abstract (English)
We study how to predict the downstream closed-loop performance of a learned latent world model from validation-time diagnostics alone. Choosing the right checkpoint from a world-model training run is difficult: validation loss and multi-step prediction RMSE keep improving long after closed-loop performance has collapsed. We present a suite of structural validation-time diagnostics drawn from optimal-control theory and apply them to Gymnasium's LunarLander v3, which features shaped rewards. We train an RSSM [5, 4] world model on it and treat per checkpoint CEM-MPC return as the oracle for closed-loop quality. By evaluating 40 metrics against this oracle, we find that the strongest single predictor is the Reward Observability Fraction (ROF), which measures the reward predictor's dependence on the observable subspace. We combine ROF with three structural regularizers into a single-number offline checkpoint selection score, the Composite Reward Observability Fraction (CROF). The CROF-selected world model trains a model-based A2C policy that beats a fairly evaluated model-free A2C baseline by ~24.5 return points while using ~65x fewer real-environment interactions, and the same world model also drives a strong zero-shot CEM-MPC policy. Code and data: https://github.com/nsmoly/LunarLander_RSSM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。