用扩散模型生成新轨迹分支，让决策Transformer摆脱次优路径。
Enhancing Decision Transformer with Diffusion-Based Trajectory Branch Generation
- 用扩散模型基于现有轨迹片段生成高回报分支
- 在D4RL上超越当前最优序列建模方法
- 适合研究离线强化学习与轨迹生成的学者
决策Transformer（DT）通过将离线强化学习转化为监督序列建模任务来学习有效策略，其中轨迹元素以回报预期（RTG）为条件自回归生成。然而，序列建模方法容易收敛到数据集中的次优轨迹，因缺乏连接更优轨迹的数据，即使条件设置为最高RTG亦如此。为此，本文提出基于扩散模型的轨迹分支生成（BG），利用扩散模型扩展数据集中的轨迹，生成基于现有轨迹片段且具有更高回报的新分支。将生成分支与原轨迹段拼接，形成扩展轨迹，使DT获得更多学习向更优轨迹转移的机会，从而避免陷入次优解。实验证明，经BG处理后，DT在D4RL基准上表现优于现有先进序列建模方法，且无需额外修改模型结构。
原文摘要 · Abstract (English)
Decision Transformer (DT) can learn effective policy from offline datasets by converting the offline reinforcement learning (RL) into a supervised sequence modeling task, where the trajectory elements are generated auto-regressively conditioned on the return-to-go (RTG).However, the sequence modeling learning approach tends to learn policies that converge on the sub-optimal trajectories within the dataset, for lack of bridging data to move to better trajectories, even if the condition is set to the highest RTG.To address this issue, we introduce Diffusion-Based Trajectory Branch Generation (BG), which expands the trajectories of the dataset with branches generated by a diffusion model.The trajectory branch is generated based on the segment of the trajectory within the dataset, and leads to trajectories with higher returns.We concatenate the generated branch with the trajectory segment as an expansion of the trajectory.After expanding, DT has more opportunities to learn policies to move to better trajectories, preventing it from converging to the sub-optimal trajectories.Empirically, after processing with BG, DT outperforms state-of-the-art sequence modeling methods on D4RL benchmark, demonstrating the effectiveness of adding branches to the dataset without further modifications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。