从训练好的Transformer中提取可解释的算法程序
Discovering Interpretable Algorithms by Decompiling Transformers to RASP
- 将Transformer重参数化为RASP程序，再通过因果干预发现最小必要子程序
- 在小规模Transformer上成功恢复出简单且可读的RASP程序
- 为模型内部实现简单算法提供了直接证据，适合关注可解释性的研究者
近期研究表明，Transformer的计算过程可在RASP编程语言族中模拟。这一发现增进了对Transformer表达能力与泛化性能的理解，尤其指出当问题存在简单RASP程序时，Transformer能精确实现长度泛化。然而，训练后的模型是否实际执行简单可解释的程序仍不清楚。本文提出一种通用方法，从训练好的Transformer中提取此类程序：首先将Transformer忠实重参数化为RASP程序，再通过因果干预识别出一个简洁的充分子程序。在多个算法与形式语言任务上训练的小型Transformer实验表明，该方法通常能恢复出简单且可解释的RASP程序。结果为变压器模型内部确实实现简单RASP程序提供了迄今为止最直接的证据。
原文摘要 · Abstract (English)
Recent work has shown that the computations of Transformers can be simulated in the RASP family of programming languages. These findings have enabled improved understanding of the expressive capacity and generalization abilities of Transformers. In particular, Transformers have been suggested to length-generalize exactly on problems that have simple RASP programs. However, it remains open whether trained models actually implement simple interpretable programs. In this paper, we present a general method to extract such programs from trained Transformers. The idea is to faithfully re-parameterize a Transformer as a RASP program and then apply causal interventions to discover a small sufficient sub-program. In experiments on small Transformers trained on algorithmic and formal language tasks, we show that our method often recovers simple and interpretable RASP programs from length-generalizing transformers. Our results provide the most direct evidence so far that Transformers internally implement simple RASP programs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。