arXiv:2511.00059cs.LGcs.AI2025-11

自动发现棋类模型中编码规则逻辑的神经元,让深层推理可解释。

Automatically Finding Rule-Based Neurons in OthelloGPT

  • 用回归决策树建模棋盘状态与神经元激活的关系。
  • 913个神经元可被规则树精准描述(R²>0.7),占层5约45%。
  • 实验证明相关神经元对预测合法走法有因果影响,适合可解释性研究者使用。

OthelloGPT 是一个用于预测奥赛罗棋合法走法的 Transformer 模型,为可解释性研究提供了理想实验平台。该模型足够复杂,能呈现丰富的计算模式,同时基于规则化的游戏逻辑,便于逆向解析。本文提出一种基于决策树的自动化方法,用于识别并解读编码规则逻辑的 MLP 神经元。通过训练回归决策树将棋盘状态映射到神经元激活值,并提取神经元高度活跃的决策路径,将其转化为人类可读的逻辑表达式。结果揭示了高度可解释的模式,例如特定神经元专门检测对角线走法是否合法。研究发现,层5中约一半神经元(2,048个中的913个)可由紧凑的规则决策树准确描述(R² > 0.7),其余可能参与更分布式或非规则化计算。通过定向干预验证了决策树识别出模式的因果重要性:针对特定棋盘模式,抑制对应神经元后,模型在这些模式上的合法走法预测能力下降约5-10倍,显著高于对照模式。为促进后续研究,本文提供了一个 Python 工具,可将规则化游戏行为映射至其对应的神经元,作为研究人员检验可解释方法是否还原真实计算结构的资源。

原文摘要 · Abstract (English)

OthelloGPT, a transformer trained to predict valid moves in Othello, provides an ideal testbed for interpretability research. The model is complex enough to exhibit rich computational patterns, yet grounded in rule-based game logic that enables meaningful reverse-engineering. We present an automated approach based on decision trees to identify and interpret MLP neurons that encode rule-based game logic. Our method trains regression decision trees to map board states to neuron activations, then extracts decision paths where neurons are highly active to convert them into human-readable logical forms. These descriptions reveal highly interpretable patterns; for instance, neurons that specifically detect when diagonal moves become legal. Our findings suggest that roughly half of the neurons in layer 5 can be accurately described by compact, rule-based decision trees ($R^2 > 0.7$ for 913 of 2,048 neurons), while the remainder likely participate in more distributed or non-rule-based computations. We verify the causal relevance of patterns identified by our decision trees through targeted interventions. For a specific square, for specific game patterns, we ablate neurons corresponding to those patterns and find an approximately 5-10 fold stronger degradation in the model's ability to predict legal moves along those patterns compared to control patterns. To facilitate future work, we provide a Python tool that maps rule-based game behaviors to their implementing neurons, serving as a resource for researchers to test whether their interpretability methods recover meaningful computational structures.

可解释性神经元分析决策树棋类模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。