用可训练的解释助手，从神经网络激活中提取关键概念并预测行为。
Predictive Concept Decoders: Training Scalable End-to-End Interpretability Assistants
- 通过压缩激活为稀疏概念列表，实现端到端可训练的解释机制。
- 在大量数据上预训练后，能准确检测越狱攻击和隐藏语义特征。
- 适合需要透明化模型决策过程的研究者与开发者使用。
解释神经网络内部激活可以生成更忠实的行为说明，但因激活空间结构复杂而困难。现有可扩展解释方法依赖人工设计的代理来假设并验证激活与外部行为的关系。我们提出将该任务转化为端到端训练目标：训练解释助手通过通信瓶颈，从激活中准确预测模型行为。具体地，编码器将激活压缩为稀疏的概念列表，解码器读取此列表并回答自然语言问题。我们展示了如何在大规模非结构化数据上预训练该助手，再微调以回答问题。所提出的架构称为预测概念解码器（Predictive Concept Decoder, PCD），具有良好的可扩展性：瓶颈概念的自解释得分随数据量增加而提升，下游应用性能也随之提高。PCDs 可检测越狱攻击、隐秘提示及植入的潜在概念，并准确揭示潜在用户属性。
原文摘要 · Abstract (English)
Interpreting the internal activations of neural networks can produce more faithful explanations of their behavior, but is difficult due to the complex structure of activation space. Existing approaches to scalable interpretability use hand-designed agents that make and test hypotheses about how internal activations relate to external behavior. We propose to instead turn this task into an end-to-end training objective, by training interpretability assistants to accurately predict model behavior from activations through a communication bottleneck. Specifically, an encoder compresses activations to a sparse list of concepts, and a decoder reads this list and answers a natural language question about the model. We show how to pretrain this assistant on large unstructured data, then finetune it to answer questions. The resulting architecture, which we call a Predictive Concept Decoder, enjoys favorable scaling properties: the auto-interp score of the bottleneck concepts improves with data, as does the performance on downstream applications. Specifically, PCDs can detect jailbreaks, secret hints, and implanted latent concepts, and are able to accurately surface latent user attributes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。