arXiv:2507.10174cs.AIcs.LG2025-07中稿 · RLBrew: Ingredient…被引 3

简单过滤数据+行为克隆,比决策Transformer更适用于稀疏奖励场景。

Should We Ever Prefer Decision Transformer for Offline Reinforcement Learning?

  • 用过滤低质量轨迹的方法改进行为克隆(FBC)
  • FBC在稀疏奖励任务上超越决策Transformer
  • 适合追求高效、少数据的离线强化学习应用

近年来,大量研究探索将Transformer架构应用于强化学习。其中,决策Transformer(DT)因能将回报条件策略学习转化为序列建模任务,在离线强化学习中备受关注。最近,Bhargava等人(2024)系统比较了DT与传统基于MLP的离线强化学习算法(如行为克隆BC和保守Q-learning CQL),声称DT在稀疏奖励和低质量数据设置下表现更优。本文在机器人抓取任务(Robomimic)和运动控制基准(D4RL)上进行实验,发现基于MLP的过滤行为克隆(FBC)在稀疏奖励环境中表现竞争性甚至更优。FBC仅需剔除低绩效轨迹后对剩余数据进行标准行为克隆,方法简单,训练数据需求少且计算效率更高。结果表明,DT在稀疏奖励环境下并非更优。结合已有研究,DT在密集奖励环境中也未必更优。因此我们提出:决策Transformer是否始终值得优先使用?

原文摘要 · Abstract (English)

In recent years, extensive work has explored the application of the Transformer architecture to reinforcement learning problems. Among these, Decision Transformer (DT) has gained particular attention in the context of offline reinforcement learning due to its ability to frame return-conditioned policy learning as a sequence modeling task. Most recently, Bhargava et al. (2024) provided a systematic comparison of DT with more conventional MLP-based offline RL algorithms, including Behavior Cloning (BC) and Conservative Q-Learning (CQL), and claimed that DT exhibits superior performance in sparse-reward and low-quality data settings. In this paper, through experimentation on robotic manipulation tasks (Robomimic) and locomotion benchmarks (D4RL), we show that MLP-based Filtered Behavior Cloning (FBC) achieves competitive or superior performance compared to DT in sparse-reward environments. FBC simply filters out low-performing trajectories from the dataset and then performs ordinary behavior cloning on the filtered dataset. FBC is not only very straightforward, but it also requires less training data and is computationally more efficient. The results therefore suggest that DT is not preferable for sparse-reward environments. From prior work, arguably, DT is also not preferable for dense-reward environments. Thus, we pose the question: Is DT ever preferable?

强化学习离线学习决策模型行为克隆

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。