通过约束策略减少分布外查询，解决模型规划中价值过估计问题。
TD-M(PC)$^2$: Improving Temporal Difference MPC Through Policy Constraint
- 引入策略正则化项，抑制规划生成数据与学习策略间的分布不匹配。
- 在61自由度人形机器人任务中，性能超越TD-MPC2显著提升。
- 无需额外计算，可无缝集成至现有模型规划框架中。
基于模型的强化学习算法结合了模型规划与学习的价值/策略先验，在连续控制任务中展现出高数据效率和优异性能。然而我们发现，依赖标准SAC风格策略迭代进行价值学习、直接使用规划器生成数据的方法，常导致持续的价值过估计。通过理论分析与实验，我们认为这一问题根源于规划器始终引导的数据生成策略与学习到的策略先验之间的结构化策略不匹配。为以最小改动缓解此不匹配，我们提出一种策略正则化项，有效减少分布外（OOD）查询，从而改善价值学习。该方法仅需对现有框架做极少修改，且无需额外计算开销。大量实验表明，所提方法在多个基准上显著优于基线（如TD-MPC2），尤其在61自由度人形机器人任务中表现突出。定性结果详见：https://darthutopian.github.io/tdmpc_square/
原文摘要 · Abstract (English)
Model-based reinforcement learning algorithms that combine model-based planning and learned value/policy prior have gained significant recognition for their high data efficiency and superior performance in continuous control. However, we discover that existing methods that rely on standard SAC-style policy iteration for value learning, directly using data generated by the planner, often result in \emph{persistent value overestimation}. Through theoretical analysis and experiments, we argue that this issue is deeply rooted in the structural policy mismatch between the data generation policy that is always bootstrapped by the planner and the learned policy prior. To mitigate such a mismatch in a minimalist way, we propose a policy regularization term reducing out-of-distribution (OOD) queries, thereby improving value learning. Our method involves minimum changes on top of existing frameworks and requires no additional computation. Extensive experiments demonstrate that the proposed approach improves performance over baselines such as TD-MPC2 by large margins, particularly in 61-DoF humanoid tasks. View qualitative results at https://darthutopian.github.io/tdmpc_square/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。