通过激活分析精准定位语言模型中影响行为的模块,提升推理时控制效果。
REAL: Reading Out Transformer Activations for Precise Localization in Language Model Steering
- 用向量量化自编码器划分隐藏层激活空间,区分与行为相关/无关部分
- 在9个数据集上平均提升20%(最高81.5%),显著优于现有方法
- 所选模块具备跨领域零样本泛化能力,适合需要精准控制的场景
推理时控制旨在不修改参数的情况下改变大语言模型的输出,但核心挑战在于识别对目标行为影响最大的内部模块。现有方法常依赖简单线索或随意启发式,导致效果不佳或产生意外结果。我们提出REAL框架,用于识别Transformer模型中与行为相关的模块(注意力头或层)。针对每个模块,REAL在其隐藏激活上训练向量量化自编码器(VQ-AE),并使用共享可学习码本将潜在空间划分为行为相关和无关子空间。通过二分类指标衡量模块的VQ-AE编码在区分行为一致与违背响应时的表现,以此量化其行为相关性,进而指导模块选择与控制强度。我们在来自Llama和Qwen家族的8个LLM及9个数据集上评估REAL,涵盖真实性增强、知识冲突下的开放域问答和通用对齐任务。REAL实现更有效的推理时干预,在真实性控制任务上相比ITI方法平均相对提升20%(最高达81.5%)。此外,由REAL选出的模块在跨领域真实性控制中展现出强零样本泛化能力。
原文摘要 · Abstract (English)
Inference-time steering aims to alter a large language model's (LLM's) responses without changing its parameters, but a central challenge is identifying the internal modules that most strongly govern the target behavior. Existing approaches often rely on simplistic cues or ad hoc heuristics, leading to suboptimal or unintended effects. We introduce REAL, a framework for identifying behavior-relevant modules (attention heads or layers) in Transformer models. For each module, REAL trains a vector-quantized autoencoder (VQ-AE) on its hidden activations and uses a shared, learnable codebook to partition the latent space into behavior-relevant and behavior-irrelevant subspaces. REAL quantifies a module's behavioral relevance by how well its VQ-AE encodings discriminate behavior-aligned from behavior-violating responses via a binary classification metric; this score guides both module selection and steering strength. We evaluate REAL across eight LLMs from the Llama and Qwen families and nine datasets spanning truthfulness enhancement, open-domain QA under knowledge conflicts, and general alignment tasks. REAL enables more effective inference-time interventions, achieving an average relative improvement of 20% (up to 81.5%) over the ITI method on truthfulness steering. In addition, the modules selected by REAL exhibit strong zero-shot generalization in cross-domain truthfulness-steering scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。