不依赖标注数据,用自然语言指令挖掘模型内部推理特征。
Unsupervised Features Mining via Activation Geometry
- 用统一指令前置输入,通过激活变化捕捉模型内部推理特征。
- 提取的特征可线性表示,通过激活操控改变模型判断结果。
- 适用于模型可解释性研究与对抗样本检测数据筛选。
现有可解释性方法通常依赖人类定义的概念标签,可能引入偏见。本文提出无监督的激活几何挖掘框架(MAG),通过在每个输入前添加相同自然语言指令(如“此物体能在沙漠中找到吗?”)来激发模型内部推理特征。通过测量指令前后单个读出点的激活差异 $m(Q ext{ }| p) - m(p)$,识别关键特征。我们探索了八种 MAG 变体,发现提取的特征能预测模型自身的世界认知与判断,且可近似为单一激活方向;部分特征线性表达更强,可通过向量引导(激活操控)改变模型决策。最后,该方法用于筛选提示注入分类器的最佳训练数据:尽管普通激活相似性与下游性能几乎无关,基于 RFD 的相似性达到 94.7% 的 Top-1 和 100% 的 Top-2 准确率。
原文摘要 · Abstract (English)
Interpretability methods aim to reveal the features represented inside large language models (LLMs). Many existing methods begin with labeled examples of a human-defined concept that may reflect human biases, and then identify how that concept is represented within the model, for example in its activation space or through other decomposition methods. We introduce \emph{Mining via Activation Geometry} (MAG), a simple unsupervised framework for extracting reasoning features from model activations by prepending the same natural-language instruction $Q$ to every input $p$, where $Q$ defines the reasoning feature of interest, such as ``Can this object be found in the desert?'' or ``Is this prompt malicious?'' We measure how the instruction changes the model's internal representation using $m(Q \mid p) - m(p)$ at a single readout point. We explore eight different MAGs. The extracted reasoning features predict the models' own world understanding and judgment, can be approximated into a single activation direction, we found that some features are more linearly represented and some less, this linear representation, which is vector steering, can change the LLMs' decisions through activation steering by injecting reasoning features. Finally, we use the same method to select the best training datasets for prompt-injection classifier probes: while similarity between ordinary activations is almost unrelated to downstream performance, RFD-based similarity achieves $94.7\%$ Top-1 and $100\%$ Top-2 accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。