用大模型自动标注语言模型的计算路径，省去人工分组耗时。
LLMs Can Annotate Attribution Graphs
- 把特征描述给大模型，让它自动合并成可理解的超级节点。
- 在97/100的提示中准确识别出中间步骤的超级节点。
- 可批量处理上千张图，筛选出值得人工复核的有趣案例。
电路追踪是揭示语言模型内部计算过程的有力方法，但需耗时的人工步骤将单个特征或MLP神经元归类为超级节点。本文提出一种简单流水线：直接将特征描述输入大模型，由其完成超级节点分组。通过自动化可解释性指标验证，该方法生成的超级节点与人工标注效果相当。在两跳首都任务中，97/100的提示中成功恢复了中间跳的超级节点。最后，我们以该流水线为基础，进行开放式探索：自动标注1000张来自维基百科提示补全的归因图，并用大模型裁判筛选出值得人工审查的有趣图。本工作表明，即使简单的自动化也能生成有意义的归因图标注，推动进一步的自动化电路追踪研究。
原文摘要 · Abstract (English)
Circuit tracing is an exciting technique for revealing the internal computation of language models, but it requires a time-intensive manual step of grouping individual features or MLP neurons into supernodes. We present a simple pipeline for automating this step: directly presenting feature descriptions to a language model that groups them into supernodes. Using automated interpretability metrics, we confirm that supernodes generated by our pipeline are as interpretable as those generated by human annotators. On a two-hop Capitals task, our pipeline recovers a supernode corresponding to the intermediate hop in 97 of 100 prompts. Finally, we present a simple proof of concept using our pipeline for open-ended exploration, where we automatically annotate 1000 attribution graphs from Wikipedia prompt completions and then use an LLM judge to flag interesting graphs worth human review. We hope this work demonstrates that even simple automation can produce meaningful attribution graph annotations, motivating further work on automated circuit tracing.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。