通过保留关键注意力模式，实现高效可控的模型行为调整。
Don't Lose Focus: Activation Steering via Key-Orthogonal Projections
- 引入关键向量正交投影，限制注意力重路由带来的副作用。
- 在多个基准上降低性能退化5-7倍，保持95%以上原有效果。
- 适合需要兼顾推理与生成质量的长文本任务场景。
激活控制通过干预模型内部表示来引导大语言模型的行为，但常导致推理与检索性能下降。我们指出，其主要原因是注意力重路由：调控向量改变了查询与键的匹配关系，使注意力偏离语境重要词元，转向信息量较低的词元。为此，我们提出基于关键向量正交投影的调控方法（SKOP），在不牺牲调控效果的前提下，抑制有害的注意力重路由。SKOP通过保留模型依赖的少量关键词元上的注意力模式，同时允许非关键尾部词元间的注意力重新分配。在多个调控基准测试中，SKOP实现了最优的调控-效用权衡，将性能退化降低5-7倍，同时保持超过95%的原始调控效能。结果还表明，在长上下文检索场景中，传统调控方法失效时，SKOP可通过避免注意力重路由维持稳定表现。
原文摘要 · Abstract (English)
Activation steering controls LLM behaviour towards target behaviour by intervening in internal representations, yet it often degrades reasoning and retrieval performance. We argue that a primary cause of this trade-off is attention rerouting: steering vectors alter query-key matching, shifting attention away from contextually important tokens toward less informative ones. To address this, we propose Steering via Key-Orthogonal Projections (SKOP), a steering method that constrains harmful attention rerouting without eliminating steering efficacy. SKOP achieves this by preserving attention patterns on a small set of focus tokens the model relies on for reasoning and retrieval, while allowing redistribution among less critical tail tokens. Across multiple steering benchmarks, we show that SKOP achieves the best joint steering-utility trade-off, reducing utility degradation by 5-7x while retaining over 95% of vanilla steering efficacy. Our results further suggest that, in long-context retrieval settings where vanilla steering approaches are ineffective, SKOP can maintain robust performance by avoiding attention rerouting.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。