发现大模型解码时共享的低维决策空间，可显著影响推理性能。
DecodeShare: Tracing the Shared Subspace of LLM Decode-Time Decisions
- 通过识别解码阶段共用的低维隐藏状态子空间，揭示任务通用结构。
- 移除该共享空间导致性能下降远超随机或预填充衍生空间。
- 适用于模型可解释性与激活操控，提升下游部署可靠性。
大型语言模型（LLMs）以统一参数处理多种任务，但在键值缓存推理中，解码阶段是否使用任务通用结构尚不明确。本文提出DecodeShare协议，识别解码阶段隐藏状态中跨任务一致共享的低维子空间，并通过仅在解码时移除该子空间来检验其因果作用。实验表明，在相同干预预算下，破坏该共享子空间导致决策性能下降程度远超破坏预填充推导或随机子空间。进一步证明该解码共享子空间对激活操控具有实际影响：常见操控方向会重叠于任务通用解码通道。投影移除该共享子空间能直接分离两个组件的功能角色，且在解码阶段评估操控向量比基于预填充的代理更可靠。尽管结构紧凑，该共享子空间仍是解码阶段高杠杆的因果通道。代码已开源：https://github.com/Zishan-Shao/decodeshare.git。
原文摘要 · Abstract (English)
Large language models (LLMs) handle many tasks with one set of parameters, but under KV-cached inference it is unclear what task-general structure, if any, is used at decode time rather than during prefill. We propose DecodeShare, a protocol that identifies a low-dimensional subspace consistently shared across tasks in decode-time hidden states, and then tests its causal role by removing that subspace only during decoding. In our experiments, disturbing the discovered shared subspace degrades decision performance far more than disturbing either a prefill-derived or random subspace under the same intervention budget. We further show this decode-shared subspace has practical consequences for activation steering: common steering directions can overlap the task-general decode channel. Projecting out this shared subspace directly separates the functional roles of the two components, while evaluating steering vectors at decode-time yields more reliable signal for downstream deployment than prefill-based proxies. Despite its compactness, the shared subspace can serve as a high-leverage causal channel at decode time. Code is available at: https://github.com/Zishan-Shao/decodeshare.git.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。