提出新方法减少模型误差累积，提升强化学习数据质量
On Rollouts in Model-Based Reinforcement Learning
- 分离随机与认知不确定性，降低后者对数据分布影响
- 跟踪误差累积并设置终止条件，有效防止数据污染
- 在MuJoCo任务上实现当前最优性能，支持更长滚动步长
基于模型的强化学习（MBRL）通过学习环境模型并生成合成轨迹来提升数据效率。然而，轨迹中累积的模型误差会扭曲数据分布，影响策略学习并阻碍长期规划，成为当前方法的主要瓶颈。本文提出Infoprop，一种基于模型的轨迹生成机制，能区分随机不确定性与认知不确定性，并减少后者对数据分布的影响。同时，Infoprop 跟踪模型轨迹中的累积误差，提供终止条件以限制数据污染。我们在 Infoprop-Dyna 算法中验证了该方法，在常见的 MuJoCo 基准任务上实现了动态风格 MBRL 的当前最优性能，显著提升轨迹长度与数据质量。
原文摘要 · Abstract (English)
Model-based reinforcement learning (MBRL) seeks to enhance data efficiency by learning a model of the environment and generating synthetic rollouts from it. However, accumulated model errors during these rollouts can distort the data distribution, negatively impacting policy learning and hindering long-term planning. Thus, the accumulation of model errors is a key bottleneck in current MBRL methods. We propose Infoprop, a model-based rollout mechanism that separates aleatoric from epistemic model uncertainty and reduces the influence of the latter on the data distribution. Further, Infoprop keeps track of accumulated model errors along a model rollout and provides termination criteria to limit data corruption. We demonstrate the capabilities of Infoprop in the Infoprop-Dyna algorithm, reporting state-of-the-art performance in Dyna-style MBRL on common MuJoCo benchmark tasks while substantially increasing rollout length and data quality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。