剖析强化学习模型内部机制，发现其导航偏见与决策基础。
Mechanistic Interpretability of Reinforcement Learning Agents
- 通过分解神经网络，识别出墙壁和路径等基础特征
- 发现代理在无目标时仍倾向朝右上角移动，存在目标误泛化
- 开发交互式激活探索工具，可视化模型决策过程
本文通过分析在程序化迷宫环境训练的神经网络，探索强化学习（RL）代理的机械可解释性。解构网络内部结构后，我们识别出迷宫墙壁和路径等基础特征，这些构成了模型决策的基础。一个重要发现是目标误泛化：即使没有明确目标，该RL代理也表现出对特定导航策略的偏好，例如始终朝向右上角移动。借助显著性映射和特征映射技术，我们可视化了这些偏差。此外，我们还开发了新型工具，支持交互式探索各层激活状态，进一步揭示模型内部运作机制。
原文摘要 · Abstract (English)
This paper explores the mechanistic interpretability of reinforcement learning (RL) agents through an analysis of a neural network trained on procedural maze environments. By dissecting the network's inner workings, we identified fundamental features like maze walls and pathways, forming the basis of the model's decision-making process. A significant observation was the goal misgeneralization, where the RL agent developed biases towards certain navigation strategies, such as consistently moving towards the top right corner, even in the absence of explicit goals. Using techniques like saliency mapping and feature mapping, we visualized these biases. We furthered this exploration with the development of novel tools for interactively exploring layer activations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。