提出高效在线动作分块强化学习方法,提升长时序任务探索与学习效率。
SEAR: Sample Efficient Action Chunking Reinforcement Learning
- 用因果Transformer critic和多时域目标,提升动作序列训练梯度利用率。
- 通过滚动窗口与随机重规划,保持决策频率并实现状态均匀覆盖。
- 在MetaWorld等复杂任务上超越SimbaV2,适合长时序在线强化学习场景。
动作分块能改善长时序强化学习中的探索效率并加速价值传播,但直接将离线策略方法应用于降低决策频率的动作扩展空间会抵消这些优势,导致样本效率低下。现有在线动作分块方法依赖计算成本高的仅评论价器方法或依赖离线数据。本文提出SEAR,一种样本高效的离线策略在线强化学习算法,同时解决上述挑战。为应对高维动作序列,SEAR采用因果Transformer评论家,结合多时域目标,为每个动作分块前缀提供训练信号,显著提升每样本有效梯度。为恢复决策频率,SEAR采用滚动视野与随机重规划策略,在保证大动作块快速价值传播的同时保留小动作块的反应能力。SEAR在具有挑战性的MetaWorld操作任务中优于当前最优在线强化学习方法(如SimbaV2)。此外,将SEAR应用于离线转在线方法QC,在OGBench cube-triple任务上进一步提升了性能。
原文摘要 · Abstract (English)
Action chunking improves exploration and accelerates value propagation in long-horizon reinforcement learning, but naively applying off-policy methods to the temporally extended action space at reduced decision frequency offsets these gains, leading to poor sample efficiency. Existing online action chunking methods address these issues through computationally expensive critic-only approaches or by relying on offline data. We introduce SEAR, a sample-efficient off-policy algorithm that enables online reinforcement learning with action chunks by addressing both challenges. To handle high-dimensional action sequences, SEAR employs a causal transformer critic trained with multi-horizon targets that provide a training signal for every prefix of an action chunk, effectively increasing the useful gradients per sample. To restore decision frequency, SEAR operates with a receding horizon and random replanning, ensuring uniform state coverage while combining the fast value propagation of large chunks with the reactivity of small ones. SEAR outperforms state-of-the-art online reinforcement learning methods including SimbaV2 on challenging Metaworld manipulation tasks. Beyond online RL, applying SEAR to the offline-to-online method QC improves its performance on the demanding OGBench cube-triple tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。