用强化学习梯度提升决策变压器在线微调效果，尤其适合低奖励预训练数据。
Reinforcement Learning Gradients as Vitamin for Online Finetuning Decision Transformers
- 引入TD3梯度替代传统返回目标，改善在线微调过程
- 在低奖励离线数据预训练下，性能显著提升超过20%
- 为决策变压器优化提供新思路，适合强化学习研究者
决策变压器作为离线强化学习的新范式，通过自回归方式完成轨迹生成。尽管已有改进，但其在线微调仍研究不足。现有主流方法ODT在低奖励离线数据预训练时表现不佳。本文理论分析表明，常用的返回目标（RTG）与期望回报差距较大，阻碍了在线微调。而标准强化学习算法中的价值函数和优势函数可有效缓解此问题。实验验证，仅在ODT微调中加入TD3梯度，即可显著提升性能，尤其在低奖励预训练场景下提升超过20%。该发现为决策变压器的进一步优化提供了新方向。
原文摘要 · Abstract (English)
Decision Transformers have recently emerged as a new and compelling paradigm for offline Reinforcement Learning (RL), completing a trajectory in an autoregressive way. While improvements have been made to overcome initial shortcomings, online finetuning of decision transformers has been surprisingly under-explored. The widely adopted state-of-the-art Online Decision Transformer (ODT) still struggles when pretrained with low-reward offline data. In this paper, we theoretically analyze the online-finetuning of the decision transformer, showing that the commonly used Return-To-Go (RTG) that's far from the expected return hampers the online fine-tuning process. This problem, however, is well-addressed by the value function and advantage of standard RL algorithms. As suggested by our analysis, in our experiments, we hence find that simply adding TD3 gradients to the finetuning process of ODT effectively improves the online finetuning performance of ODT, especially if ODT is pretrained with low-reward offline data. These findings provide new directions to further improve decision transformers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。