用Transformer并行生成未来状态，让强化学习搜索提速11倍
TransZero: Parallel Tree Expansion in MuZero using Transformer Networks
- 用Transformer一次性生成多个未来状态，突破传统逐层扩展瓶颈
- 在MiniGrid和LunarLander上比MuZero快11倍，仍保持高效采样
- 适合需要实时决策的复杂环境强化学习任务
我们提出TransZero，一种基于模型的强化学习算法，旨在消除蒙特卡洛树搜索（MCTS）中的串行瓶颈。与MuZero通过循环动态模型逐步构建搜索树不同，TransZero采用基于Transformer的网络，可同时生成多个潜在未来状态。结合均值-方差约束（MVC）评估器，该方法消除了对固有串行访问次数的依赖，实现了规划过程中整个子树的并行扩展。在MiniGrid和LunarLander上的实验表明，TransZero相比MuZero在真实运行时间上最高提升11倍，同时保持样本效率。结果表明，并行树结构构建可显著加速基于模型的强化学习，使复杂环境中的实时决策更接近实际应用。代码已公开于GitHub。
原文摘要 · Abstract (English)
We present TransZero, a model-based reinforcement learning algorithm that removes the sequential bottleneck in Monte Carlo Tree Search (MCTS). Unlike MuZero, which constructs its search tree step by step using a recurrent dynamics model, TransZero employs a transformer-based network to generate multiple latent future states simultaneously. Combined with the Mean-Variance Constrained (MVC) evaluator that eliminates dependence on inherently sequential visitation counts, our approach enables the parallel expansion of entire subtrees during planning. Experiments in MiniGrid and LunarLander show that TransZero achieves up to an eleven-fold speedup in wall-clock time compared to MuZero while maintaining sample efficiency. These results demonstrate that parallel tree construction can substantially accelerate model-based reinforcement learning, bringing real-time decision-making in complex environments closer to practice. The code is publicly available on GitHub.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。