揭示强化学习网络在推箱子游戏中如何用通道存储未来动作计划
Path Channels and Plan Extension Kernels: a Mechanistic Description of Planning in a Sokoban RNN
- 通过分析隐藏状态通道，发现网络用特定通道存储未来移动方向
- 卷积核编码动作导致的位置变化，构成学习到的转移模型部分
- 双向扩展机制实现类似回溯的规划，适合研究可解释性与算法理解
我们对一个基于无模型强化学习训练的卷积循环神经网络（RNN）进行部分逆向工程，使其能够玩推箱子游戏。发现该RNN将未来动作（即计划）以特定隐藏状态通道中的激活值形式存储，这些通道被称为路径通道。某个位置的高激活值表示当箱子位于该位置时，将按通道指定方向被推动。我们分析路径通道间的卷积核，发现它们编码了每个可能动作带来的位置变化，因此代表了学习到的部分转移模型。RNN通过从箱子和目标出发构建计划：这些核从前向传播路径通道激活值，从后向反向传播激活值。障碍物所在位置的通道中设置负值，使扩展核反向传播负值，从而修剪最后几步并让替代计划浮现；这是一种回溯形式。本工作表明，对计划表征的精确理解，使得我们可以用更熟悉的术语直接解读无模型训练中学习到的双向规划算法。
原文摘要 · Abstract (English)
We partially reverse-engineer a convolutional recurrent neural network (RNN) trained with model-free reinforcement learning to play the box-pushing game Sokoban. We find that the RNN stores future moves (plans) as activations in particular channels of the hidden state, which we call path channels. A high activation in a particular location means that, when a box is in that location, it will get pushed in the channel's assigned direction. We examine the convolutional kernels between path channels and find that they encode the change in position resulting from each possible action, thus representing part of a learned transition model. The RNN constructs plans by starting at the boxes and goals. These kernels extend activations in path channels forwards from boxes and backwards from the goal. Negative values are placed in channels at obstacles. This causes the extension kernels to propagate the negative value in reverse, thus pruning the last few steps and letting an alternative plan emerge; a form of backtracking. Our work shows that, a precise understanding of the plan representation allows us to directly understand the bidirectional planning-like algorithm learned by model-free training in more familiar terms.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。