无需人类数据，自动发现游戏策略选项，显著提升强化学习表现。
OptionZero: Planning with Learned Options
- 将选项网络融入MuZero，通过自对弈自主发现有效动作序列。
- 在26个Atari游戏中，平均得分提升131.58%（人类归一化）。
- 适合研究高效决策、自主策略发现的强化学习方向。
基于选项的规划——即一系列基础动作的组合——已被证明在复杂环境中的强化学习中非常有效。以往研究多依赖预定义选项或通过专家示范数据学习选项。受MuZero启发，其能在无任何人类知识的情况下学习超人级启发式策略，我们提出一种新方法OptionZero。该方法将选项网络引入MuZero，通过自对弈游戏实现选项的自主发现。同时，我们修改动态网络，在使用选项时提供环境状态转移，使在相同模拟约束下能探索更深。在26个Atari游戏上的实验证明，OptionZero优于MuZero，平均人类归一化得分提升131.58%。行为分析显示，OptionZero不仅学会选项，还掌握针对不同游戏特征的战略技能。研究结果为规划中选项的发现与利用提供了有前景的方向。代码已公开于https://rlg.iis.sinica.edu.tw/papers/optionzero。
原文摘要 · Abstract (English)
Planning with options -- a sequence of primitive actions -- has been shown effective in reinforcement learning within complex environments. Previous studies have focused on planning with predefined options or learned options through expert demonstration data. Inspired by MuZero, which learns superhuman heuristics without any human knowledge, we propose a novel approach, named OptionZero. OptionZero incorporates an option network into MuZero, providing autonomous discovery of options through self-play games. Furthermore, we modify the dynamics network to provide environment transitions when using options, allowing searching deeper under the same simulation constraints. Empirical experiments conducted in 26 Atari games demonstrate that OptionZero outperforms MuZero, achieving a 131.58% improvement in mean human-normalized score. Our behavior analysis shows that OptionZero not only learns options but also acquires strategic skills tailored to different game characteristics. Our findings show promising directions for discovering and using options in planning. Our code is available at https://rlg.iis.sinica.edu.tw/papers/optionzero.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。