用大模型生成可解释性测试集，解决模型权重与功能对应难题。
TracrBench: Generating Interpretability Testbeds with Large Language Models
- 用大模型自动生成带真实映射的RASP程序，降低人工成本。
- 构建包含121个程序的TracrBench数据集，含人工验证的权重与代码。
- 揭示当前大模型生成复杂逻辑程序能力有限，适合评估可解释性方法。
理解基于Transformer的语言模型机制仍是开放挑战，尤其因参数量庞大且缺乏权重与功能间的真值映射，制约了可解释性方法的评估。为应对这一问题，研究提出在RASP中编译具备内在真值映射的Tracr方法。然而，手动构建足够多的模型耗时费力。本文提出一种新方法，利用大语言模型（LLM）自动生成可解释性测试集，并构建了TracrBench数据集，包含121个经人工编写和大模型生成、且经人工验证的RASP程序及其对应的Transformer权重。实验发现，尽管使用GPT-4-turbo进行20次提示并采用最佳5次采样，仅正确实现101个测试程序中的57个，仍需人工补全剩余部分。该数据集旨在为可解释性方法提供有效评估基准。
原文摘要 · Abstract (English)
Achieving a mechanistic understanding of transformer-based language models is an open challenge, especially due to their large number of parameters. Moreover, the lack of ground truth mappings between model weights and their functional roles hinders the effective evaluation of interpretability methods, impeding overall progress. Tracr, a method for generating compiled transformers with inherent ground truth mappings in RASP, has been proposed to address this issue. However, manually creating a large number of models needed for verifying interpretability methods is labour-intensive and time-consuming. In this work, we present a novel approach for generating interpretability test beds using large language models (LLMs) and introduce TracrBench, a novel dataset consisting of 121 manually written and LLM-generated, human-validated RASP programs and their corresponding transformer weights. During this process, we evaluate the ability of frontier LLMs to autonomously generate RASP programs and find that this task poses significant challenges. GPT-4-turbo, with a 20-shot prompt and best-of-5 sampling, correctly implements only 57 out of 101 test programs, necessitating the manual implementation of the remaining programs. With its 121 samples, TracrBench aims to serve as a valuable testbed for evaluating and comparing interpretability methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。