用动作块提升离线模型强化学习的长程任务表现
Scalable Offline Model-Based RL with Action Chunks
- 用动作块替代单步动作预测，减少模型误差累积
- 在1亿条数据上实验，长周期任务性能领先
- 适合处理复杂、长期依赖的离线强化学习问题
本文研究模型基强化学习（特别是基于模型的价值扩展）能否为复杂、长周期任务提供可扩展的解决方案。模型基价值扩展通过当前策略和学习到的动力学模型生成长度为n的虚拟轨迹来拟合在线策略价值函数。虽然更大的n能降低价值自举偏差，但会放大长周期中的累积模型误差，导致未来预测退化。为此，我们提出一种动作块模型，该模型从一段动作序列（即“动作块”）预测未来状态，而非单个动作，从而减少误差累积。此外，不直接训练策略以最大化奖励，而是从一个表达能力强的动作块行为策略中进行拒绝采样，防止模型被分布外动作误导。我们称此方法为【基于动作块的模型强化学习（MAC）】。在包含高达1亿条转换数据的高难度任务上进行实验，结果表明，相较于其他离线模型基强化学习算法，MAC在长周期任务中表现最佳。
原文摘要 · Abstract (English)
In this paper, we study whether model-based reinforcement learning (RL), in particular model-based value expansion, can provide a scalable recipe for tackling complex, long-horizon tasks in offline RL. Model-based value expansion fits an on-policy value function using length-n imaginary rollouts generated by the current policy and a learned dynamics model. While larger n reduces bias in value bootstrapping, it amplifies accumulated model errors over long horizons, degrading future predictions. We address this trade-off with an \emph{action-chunk} model that predicts a future state from a sequence of actions (an "action chunk") instead of a single action, which reduces compounding errors. In addition, instead of directly training a policy to maximize rewards, we employ rejection sampling from an expressive behavioral action-chunk policy, which prevents model exploitation from out-of-distribution actions. We call this recipe \textbf{Model-Based RL with Action Chunks (MAC)}. Through experiments on highly challenging tasks with large-scale datasets of up to 100M transitions, we show that MAC achieves the best performance among offline model-based RL algorithms, especially on challenging long-horizon tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。