根据输入动态选择干预层,让大模型对齐更精准。
Where to Steer: Input-Dependent Layer Selection for Steering Improves LLM Alignment

- 根据输入自适应选择最佳干预层,而非固定某一层。
- 在多类模型和行为上,性能优于固定层方法。
- 适合需要精细控制模型行为的研究者使用。
定向向量已成为一种轻量且高效的大语言模型(LLM)对齐方法,可在推理阶段通过将模型表示推向目标行为来调节模型行为。然而,现有方法通常在全局固定的层应用定向向量,隐含假设最优干预层对所有输入都相同。我们指出这一假设本质受限:不同输入中与目标行为相关的表征可能分布在不同层。理论上,我们证明不同输入需在不同层进行干预才能实现理想对齐;实证也表明,实际中最优干预层在输入间差异显著。为此,我们提出 Where to Steer(W2S),通过学习输入嵌入到最优干预层的映射,实现输入依赖的层选择。在多个大模型和对齐行为上,W2S 均一致超越固定层基线,在分布内与分布外设置下均有提升。研究强调了输入依赖控制在大模型对齐中的重要性,并表明自适应层选择是当前定向向量方法中缺失的关键设计维度。
原文摘要 · Abstract (English)
Steering vectors have emerged as a lightweight and effective approach for aligning large language models (LLMs) at inference time, enabling modulation over model behaviors by shifting LLM representations towards a target behavior. However, existing methods typically apply steering vectors at a globally fixed layer, implicitly assuming that the optimal intervention layer is invariant across inputs. We argue that this assumption is fundamentally limited, as representations relevant to a target behavior can be encoded at different layers depending on the input. Theoretically, we show that different inputs can require steering at different layers to achieve alignment with a desirable model behavior. We also provide empirical evidence that the optimal steering layer varies substantially across inputs in practice. Motivated by these observations, we introduce Where to Steer (W2S), a framework that adaptively selects the intervention layer conditioned on the input, by learning a mapping from input embeddings to optimal steering layers. Across multiple LLMs and alignment behaviors, W2S consistently outperforms fixed-layer baselines, with improvements in both in-distribution and out-of-distribution settings. Our findings highlight the importance of input-dependent control in LLM alignment and demonstrate that adaptive layer selection is a key design dimension missing in the current methodology of steering vectors.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。