用状态嵌入注入回报目标，让决策变压器更高效更强
Beyond Autoregressive RTG: Conditioning via Injection Outside Sequential Modeling in Decision Transformer
- 将回报目标信息注入状态向量，不再作为序列令牌
- 序列长度减少三分之一，推理速度显著提升
- 在D4RL上表现优于标准DT，接近顶尖方法
决策变压器（DT）将离线强化学习建模为自回归序列预测，通过从回报目标（RTG）、状态和动作标记的序列中预测动作取得良好效果。然而，RTG是标量，信息量远低于典型的状态或动作向量，却占用相同的计算资源。此外，Transformer的自注意力开销随序列长度呈平方增长，将RTG作为独立标记引入会增加不必要的计算负担。本文提出SlimDT，移除自回归序列中的RTG，转而在序列建模前将RTG信息注入状态表示中，使Transformer仅处理紧凑的（状态，动作）序列。这使序列长度减少三分之一，直接提升推理效率。在D4RL基准上，SlimDT在各类任务中均超越标准DT，性能接近现有最先进方法。将稀疏的条件信号与信息丰富的序列解耦，同时带来计算优势与更高任务表现。
原文摘要 · Abstract (English)
Decision Transformer (DT) formulates offline reinforcement learning as autoregressive sequence modeling, achieving promising results by predicting actions from a sequence of Return-to-Go (RTG), state, and action tokens. However, RTG is a scalar that summarizes future rewards, containing far less information than typical state or action vectors, yet it consumes the same computational budget per token. Worse, the self-attention cost of Transformers grows quadratically with sequence length, so including RTG as a separate token adds unnecessary overhead. We propose SlimDT, which removes RTG from the autoregressive sequence. Instead, we inject RTG information into the state representations before the sequential modeling step, allowing the Transformer to process only a compact (state, action) sequence. This reduces the sequence length by one-third, directly improving inference efficiency. On the D4RL benchmark, SlimDT surpasses standard DT across various tasks and achieves performance comparable to existing state-of-the-art methods. Decoupling a sparse conditioning signal from an information-rich sequence thus yields both computational gains and higher task performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。