让大模型直接解读激活值为自然语言，更精准理解自身行为。
LatentQA: Teaching LLMs to Decode Activations Into Natural Language
- 设计可输出自然语言的探测器，实现对模型激活值的开放式问答。
- 在隐藏提示词识别等任务上超越传统探针方法，准确率显著提升。
- 支持模型行为控制与可扩展性验证，适合模型可解释性研究者使用。
自上而下的透明性分析通常依赖标量或单标记输出的探测器，限制了可捕捉行为的范围。为解决此问题，我们开发了一种更具表达力的探测器，可直接输出自然语言，完成潜变量问答（LatentQA）：即针对模型激活值回答开放性问题。构建此类探测器的关键挑战在于获取激活值与自然语言描述之间的数据集。为此，我们提出一种生成激活值与对应问答对数据集的方法，并开发了微调策略，使解码大模型在此数据集上进行训练。随后，我们通过评估解码器在读取和控制目标模型激活值方面的保真度来验证其有效性。首先，在若干具有已知答案的监督阅读任务中（如揭示隐藏系统提示、关系知识提取），解码器表现优于现有探针基线。其次，证明解码器具备足够精度，可引导目标模型展现出训练期间未见的行为。最后，展示LatentQA在数据集规模与模型规模增大时仍具有良好可扩展性。
原文摘要 · Abstract (English)
Top-down transparency typically analyzes language model activations using probes with scalar or single-token outputs, limiting the range of behaviors that can be captured. To alleviate this issue, we develop a more expressive probe that can directly output natural language, performing LatentQA: the task of answering open-ended questions about activations. A key difficulty in developing such a probe is collecting a dataset mapping activations to natural-language descriptions. In response, we propose an approach for generating a dataset of activations and associated question-answer pairs and develop a fine-tuning method for training a decoder LLM on this dataset. We then validate our decoder's fidelity by assessing its ability to read and control model activations. First, we evaluate the decoder on a number of supervised reading tasks with a known answer, such as uncovering hidden system prompts and relational knowledge extraction, and observe that it outperforms competitive probing baselines. Second, we demonstrate that the decoder is precise enough to steer the target model to exhibit behaviors unseen during training. Finally, we show that LatentQA scales well with increasing dataset and model size.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。