发现角色提示中人格与任务可线性分解，但无法压缩为单一向量。
As X, Do Y: How Persona and Task Combine in Instruction-Tuned LLMs
- 在残差流特定位置，人格与任务通过正交方向叠加影响输出。
- 替换残差后生成结果与原输出差异极小（KL接近0），保留人格特征。
- 尽管局部可加，但完整行为依赖全局上下文，无法通过单点向量压缩。
角色提示 As X, do Y 在 Gemma-2-2B-IT 与 Qwen-2.5-{1.5B, 3B}-Instruct 模型中，于早期/中期层的提示-回答转换节点（最后一个提示词与前两个生成词）展现出清晰的线性分解结构：人格与任务通过部分正交的加法方向贡献。构建纯人格效应 Δ_X 与纯任务效应 Δ_Y，用 h_{BB} + Δ_X + Δ_Y 替换干净残差，可在 12 格短网格与 48 格长人格网格上，使下游输出的 KL 距离保持极小，同时保留人格特异性行为标记。自然推论是角色提示可压缩为单个缓存残差向量，但实验表明这不可行。将预测的加法缓存或理想干净残差 h_{XY} 注入去除人格文本的基线提示中，无论在单一位置还是多层，均无法逼近原始长人格目标。人格条件下的多词生成通过注意力回传至提示中的人格文本位置，单点残差无法复现此机制。残差流中的局部可加性不意味着提示可压缩。该加法结构支持对人格或任务贡献的可解释与细粒度调控；完整延续中的人格行为依赖分布式的提示/KV 机制，非局部激活算术所能替代。
原文摘要 · Abstract (English)
Role prompts of the form As X, do Y admit a clean linear decomposition at one specific site in the residual stream: the prompt-to-answer transition -- the last prompt token together with the first two generated tokens -- in an early/mid layer band. There, persona and task contribute through partially orthogonal additive directions. Forming a pure persona effect $Δ_X$, a pure task effect $Δ_Y$, and substituting $h_{BB} + Δ_X + Δ_Y$ for the clean residual yields downstream output within a small KL of clean on Gemma-2-2B-IT and Qwen-2.5-\{1.5B, 3B\}-Instruct, across a 12-cell short grid and a 48-cell long-persona grid, with persona-specific behavioral markers preserved. The natural inference from this additive structure is that the role prompt can be compressed into a single cached residual vector. \emph{We show it cannot.} Injecting the cached additive prediction -- or even the oracle clean residual $h_{XY}$ -- into a baseline host prompt with the persona text removed does not approach the clean long-persona target, at one site or at many layers. Persona-conditioned multi-token generation flows through attention back to the persona-text positions throughout the prompt, which no residual at one site reproduces. Local additivity in the residual stream does not imply prompt compressibility. The additive structure at the prompt-to-answer transition supports interpretability and fine-grained steering of persona or task contributions; persona-conditioned behavior across the full continuation depends on a distributed prompt/KV mechanism that local activation arithmetic does not displace.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。