通过误差抑制与高效更新，提升连续控制的样本效率。
EfficientTDMPC: Improved MPC Objectives for Sample-Efficient Continuous Control
- 用模型集成与深度平均降低预测误差
- 引入不确定惩罚，避开高风险动作
- 适合低数据场景下的强化学习应用
我们提出 EfficientTDMPC，一种基于 TD-MPC 算法家族的高效模型预测控制方法，用于连续控制任务。该方法通过两个关键改进减少误差：首先，采用动态模型集合，并在不同回溯深度上平均回报估计；其次，在规划目标中加入不确定性惩罚，使策略避免返回值不确定的动作。此外，通过提升经验回放缓冲区的新鲜度和降低计算开销，进一步优化性能。实验表明，在每个基准的低数据条件下，EfficientTDMPC 在 HumanoidBench-Hard 与 DMC hard 上实现当前最优的样本效率，且在 DMC easy 上达到同等水平。
原文摘要 · Abstract (English)
We introduce EfficientTDMPC, a sample-efficient model-based reinforcement learning method for continuous control built on the TD-MPC family of algorithms. Central to this family is a planner that aims to find an action sequence that maximizes the estimated return. The return is estimated using a learned model and value networks, each of which can introduce error. EfficientTDMPC proposes to reduce this error in two ways. First, it introduces an ensemble of dynamics models and averages the return estimates across those models and across different rollout depths. Second, it adds the option to apply an uncertainty penalty to the planner objective, yielding a planner that avoids actions with uncertain return estimates. It then adds practical improvements which increase buffer data freshness and reduce compute. Lastly, we find that our contributions enable EfficientTDMPC to benefit more from a higher update-to-data (UTD) ratio, further improving sample efficiency. To the best of our knowledge, in the low data regime of each benchmark, EfficientTDMPC achieves state-of-the-art (SOTA) in terms of sample efficiency on HumanoidBench-Hard and DMC hard, while matching SOTA on DMC easy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。