通过反向生成自然语言，揭示大模型内部激活的语义信息。
InverseScope: Scalable Activation Inversion for Interpreting Large Language Models
- 不依赖线性或稀疏假设，用输入反演解析模型激活。
- 在140亿参数模型上高效生成激活相近的输入。
- 可发现句子级线性类比，适合模型可解释性研究者。
理解大语言模型（LLM）的内部表示是可解释性研究的核心挑战。现有特征可解释方法常依赖线性或稀疏等强结构假设，可能与实际不符。本文提出 InverseScope，一种低假设、可扩展的神经激活反演框架。给定目标激活，该框架通过生成产生相近激活的自然语言输入，将抽象内部状态映射到具体语言层面。为克服高维激活空间采样成本高的问题，我们设计了一种新型控制层条件架构，显著提升采样效率，优于传统令牌前置方法。实验表明，InverseScope揭示了 LLM 表示空间中丰富的几何结构，包括句子级线性类比。该框架可扩展至最大140亿参数的开源模型，并能泛化到分布外输入，支持对激活邻域的系统性分析。
原文摘要 · Abstract (English)
Understanding the internal representations of large language models (LLMs) is a central challenge in interpretability research. Existing feature interpretability methods often rely on strong structural assumptions--such as linearity or sparsity--that may not hold in practice. In this work, we introduce InverseScope, an assumption-light and scalable framework for interpreting neural activations via input inversion. Given a target activation, InverseScope characterizes its encoded information by generating natural-language inputs that produce nearby activations, grounding abstract internal states in concrete language. To overcome the prohibitive cost of sampling in high-dimensional activation spaces, we propose a novel control-layer conditioning architecture that substantially improves sample efficiency compared to prior token-prepending approaches. We demonstrate that InverseScope reveals rich geometric structure in LLM representation spaces, including sentence-level linear analogies. The framework scales to state-of-the-art open-source models of up to 14B parameters and generalizes to out-of-distribution inputs, enabling systematic analysis of activation neighborhoods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。