将Transformer用于在线连续控制强化学习，发现稳定训练策略可实现优异性能。
A New Perspective on Transformers in Online Reinforcement Learning for Continuous Control
- 设计输入条件、共享网络组件与序列数据切片方式，提升Transformer稳定性
- 在全观与部分可观测任务中均达到竞争性表现，跨向量与图像场景适用
- 为在线无模型强化学习中的Transformer应用提供实用指导
尽管在离线或基于模型的强化学习中表现优异，变压器（transformers）在在线无模型强化学习中仍鲜有研究，主要因其对训练设置和模型设计（如策略与价值网络结构、组件共享、时序信息处理）敏感。本文表明，通过合理设计，transformer 可成为连续控制任务中在线无模型强化学习的有力基线。我们探究了关键设计问题：如何设置输入条件、在行动者与评论家间共享组件，以及如何切分序列数据进行训练。实验揭示了在多种任务中稳定的架构与训练策略，使模型在完全可观测与部分可观测任务中均表现良好，并适用于向量与图像输入场景。这些发现为在在线强化学习中应用变压器提供了切实可行的指导。
原文摘要 · Abstract (English)
Despite their effectiveness and popularity in offline or model-based reinforcement learning (RL), transformers remain underexplored in online model-free RL due to their sensitivity to training setups and model design decisions such as how to structure the policy and value networks, share components, or handle temporal information. In this paper, we show that transformers can be strong baselines for continuous control in online model-free RL. We investigate key design questions: how to condition inputs, share components between actor and critic, and slice sequential data for training. Our experiments reveal stable architectural and training strategies enabling competitive performance across fully and partially observable tasks, and in both vector- and image-based settings. These findings offer practical guidance for applying transformers in online RL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。