用动作序列预测回报，提升强化学习数据效率
Coarse-to-fine Q-Network with Action Sequence for Data-Efficient Reinforcement Learning
- 基于动作序列构建分层价值网络，显式建模动作组合的长期影响
- 在BiGym和RLBench上，稀疏奖励任务性能超越多个基线方法
- 适合需要少样本训练的机器人控制场景，如人体模型和桌面操作
近期行为克隆算法的成功依赖于对动作序列的预测。我们观察到，在预测真实回报时引入动作序列可降低验证损失，由此提出一种新型基于值函数的强化学习算法——分层细化动作序列Q网络（CQN-AS）。该算法通过学习输出动作序列上的Q值，显式训练价值函数以理解执行动作序列的后果。实验表明，CQN-AS在BiGym和RLBench的多种稀疏奖励人形控制与桌面操作任务中，均优于多个基线方法。
原文摘要 · Abstract (English)
Predicting a sequence of actions has been crucial in the success of recent behavior cloning algorithms in robotics. Can similar ideas improve reinforcement learning (RL)? We answer affirmatively by observing that incorporating action sequences when predicting ground-truth return-to-go leads to lower validation loss. Motivated by this, we introduce Coarse-to-fine Q-Network with Action Sequence (CQN-AS), a novel value-based RL algorithm that learns a critic network that outputs Q-values over a sequence of actions, i.e., explicitly training the value function to learn the consequence of executing action sequences. Our experiments show that CQN-AS outperforms several baselines on a variety of sparse-reward humanoid control and tabletop manipulation tasks from BiGym and RLBench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。