arXiv:2410.08979cs.LGcs.AI2024-10ICLR

让强化学习在慢决策下仍能高效控制,实现低频稳定执行。

Overcoming Slow Decision Frequencies in Continuous Control: Model-Based Sequence Reinforcement Learning for Model-Free Control

  • 用分时架构结合模型与智能体,生成动作序列以应对低频决策。
  • 在连续控制任务中性能接近顶尖算法,样本复杂度显著降低。
  • 适合需灵活决策频率的现实场景,如机器人控制或工业自动化。

强化学习(RL)正快速达到甚至超越人类水平的控制能力。然而,当前先进算法通常需要远快于人类反应速度的时间步和响应频率,在真实场景中不切实际,常需专用硬件支持。我们提出序列强化学习(SRL),一种可为给定状态生成动作序列的算法,使低频决策下的有效控制成为可能。SRL通过在不同时间尺度上运行模型与演员-评论家架构,解决动作序列学习难题。提出“时间回溯”机制:评论家利用模型预测原始动作之间的中间状态,为序列中每个动作提供学习信号。训练完成后,演员可独立生成动作序列,实现无需模型的低频控制。我们在一系列连续控制任务上评估SRL,结果表明其性能媲美先进算法,同时大幅降低演员采样复杂度。为更全面评估不同决策频率下的表现,我们引入频率平均得分(FAS)指标。实验显示,SRL在FAS上显著优于传统算法,特别适用于需变频决策的应用。此外,将SRL与基于模型的在线规划对比,发现两者在相同模型训练下获得相近的FAS表现。

原文摘要 · Abstract (English)

Reinforcement learning (RL) is rapidly reaching and surpassing human-level control capabilities. However, state-of-the-art RL algorithms often require timesteps and reaction times significantly faster than human capabilities, which is impractical in real-world settings and typically necessitates specialized hardware. We introduce Sequence Reinforcement Learning (SRL), an RL algorithm designed to produce a sequence of actions for a given input state, enabling effective control at lower decision frequencies. SRL addresses the challenges of learning action sequences by employing both a model and an actor-critic architecture operating at different temporal scales. We propose a "temporal recall" mechanism, where the critic uses the model to estimate intermediate states between primitive actions, providing a learning signal for each individual action within the sequence. Once training is complete, the actor can generate action sequences independently of the model, achieving model-free control at a slower frequency. We evaluate SRL on a suite of continuous control tasks, demonstrating that it achieves performance comparable to state-of-the-art algorithms while significantly reducing actor sample complexity. To better assess performance across varying decision frequencies, we introduce the Frequency-Averaged Score (FAS) metric. Our results show that SRL significantly outperforms traditional RL algorithms in terms of FAS, making it particularly suitable for applications requiring variable decision frequencies. Furthermore, we compare SRL with model-based online planning, showing that SRL achieves comparable FAS while leveraging the same model during training that online planners use for planning.

强化学习连续控制低频决策序列动作

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。