用蒙特卡洛束搜索提升连续控制的探索效率
Monte Carlo Beam Search for Actor-Critic Reinforcement Learning in Continuous Control
- 在策略输出附近生成多个候选动作，通过短程模拟评估选择最优
- 20万步内达90%最大奖励，比次优方法快一倍
- 适合需要高效探索的复杂连续控制任务
基于策略-评价者的方法（如TD3）依赖基础噪声探索，可能导致策略收敛不佳。本文提出蒙特卡洛束搜索（MCBS），将束搜索与蒙特卡洛滚动回放结合，用于改进探索与动作选择。MCBS在策略输出附近生成多个候选动作，并通过短时域滚动回放评估，使智能体做出更明智决策。我们在HalfCheetah-v4、Walker2d-v5和Swimmer-v5等连续控制基准上测试，结果表明，相比标准TD3及其他基线方法（如SAC、PPO、A2C），MCBS显著提升了样本效率与性能。实验显示，MCBS在不同环境中均实现更快收敛：例如，在约20万时间步内达到最大奖励的90%，而次优方法需40万时间步。我们还分析了束宽、滚动深度等关键超参数，并探索自适应策略以优化复杂控制任务表现。
原文摘要 · Abstract (English)
Actor-critic methods, like Twin Delayed Deep Deterministic Policy Gradient (TD3), depend on basic noise-based exploration, which can result in less than optimal policy convergence. In this study, we introduce Monte Carlo Beam Search (MCBS), a new hybrid method that combines beam search and Monte Carlo rollouts with TD3 to improve exploration and action selection. MCBS produces several candidate actions around the policy's output and assesses them through short-horizon rollouts, enabling the agent to make better-informed choices. We test MCBS across various continuous-control benchmarks, including HalfCheetah-v4, Walker2d-v5, and Swimmer-v5, showing enhanced sample efficiency and performance compared to standard TD3 and other baseline methods like SAC, PPO, and A2C. Our findings emphasize MCBS's capability to enhance policy learning through structured look-ahead search while ensuring computational efficiency. Additionally, we offer a detailed analysis of crucial hyperparameters, such as beam width and rollout depth, and explore adaptive strategies to optimize MCBS for complex control tasks. Our method shows a higher convergence rate across different environments compared to TD3, SAC, PPO, and A2C. For instance, we achieved 90% of the maximum achievable reward within around 200 thousand timesteps compared to 400 thousand timesteps for the second-best method.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。