简化决策变压器结构，用最新回报目标提升效率与性能
Decoupling Return-to-Go for Efficient Decision Transformer
- 仅用最新回报目标引导动作预测，不再输入全程回报序列
- 在多个离线强化学习任务中表现优于原版决策变压器
- 计算量更低，适合资源受限场景下的高效决策应用
决策变压器（DT）通过将回报目标（RTG）作为条件，在离线强化学习中展现出强大的序列建模能力。然而，我们发现该设计存在关键冗余：整个序列的RTG输入对动作预测并无必要，因为只有最新的RTG才影响决策。实验表明，这种冗余会降低模型性能。为此，我们提出解耦决策变压器（DDT），仅通过Transformer处理观测与动作序列，由最新RTG引导动作生成。该简化架构显著提升性能并降低计算开销。实验显示，DDT在多个离线强化学习任务上优于原始DT，并达到当前最优DT变体的竞争力水平。
原文摘要 · Abstract (English)
The Decision Transformer (DT) has established a powerful sequence modeling approach to offline reinforcement learning. It conditions its action predictions on Return-to-Go (RTG), using it both to distinguish trajectory quality during training and to guide action generation at inference. In this work, we identify a critical redundancy in this design: feeding the entire sequence of RTGs into the Transformer is theoretically unnecessary, as only the most recent RTG affects action prediction. We show that this redundancy can impair DT's performance through experiments. To resolve this, we propose the Decoupled DT (DDT). DDT simplifies the architecture by processing only observation and action sequences through the Transformer, using the latest RTG to guide the action prediction. This streamlined approach not only improves performance but also reduces computational cost. Our experiments show that DDT significantly outperforms DT and establishes competitive performance against state-of-the-art DT variants across multiple offline RL tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。