用规划令牌提升长时序强化学习性能，减少误差累积。
Planning Transformer: Long-Horizon Offline Reinforcement Learning with Planning Tokens
- 引入规划令牌，捕获未来长期信息以指导策略
- 在D4RL复杂环境中实现新最佳性能，显著降低误差累积
- 通过注意力图与可视化提升策略可解释性，适合长时序任务研究者
基于监督学习的离线强化学习方法，特别是决策变换器，在连续环境和稀疏奖励任务中表现良好。然而，其自回归模型在长时序任务中常因误差累积而失效。为此，本文突破传统逐项预测，引入包含高阶长期信息的规划令牌。在固定间隔预测双时间尺度令牌，使模型能利用这些规划令牌作为隐式规划，指导底层策略并减少误差累积。该结构改进显著提升了复杂D4RL环境中的长时序任务表现，达到新基准。此外，规划令牌还通过可解释的计划可视化与注意力图，增强了策略的可解释性。
原文摘要 · Abstract (English)
Supervised learning approaches to offline reinforcement learning, particularly those utilizing the Decision Transformer, have shown effectiveness in continuous environments and for sparse rewards. However, they often struggle with long-horizon tasks due to the high compounding error of auto-regressive models. To overcome this limitation, we go beyond next-token prediction and introduce Planning Tokens, which contain high-level, long time-scale information about the agent's future. Predicting dual time-scale tokens at regular intervals enables our model to use these long-horizon Planning Tokens as a form of implicit planning to guide its low-level policy and reduce compounding error. This architectural modification significantly enhances performance on long-horizon tasks, establishing a new state-of-the-art in complex D4RL environments. Additionally, we demonstrate that Planning Tokens improve the interpretability of the model's policy through the interpretable plan visualisations and attention map.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。