发现大模型任务识别靠分布式模板，单一位置干预无效。
Single-Position Intervention Fails: Distributed Output Templates Drive In-Context Learning
- 任务身份编码是分布式的，非单一位置决定。
- 多位置同时干预可实现96%任务迁移成功率。
- 适合研究模型机制与提示学习的学者阅读。
理解大语言模型如何从少量示例中编码任务身份，是机制可解释性中的核心开放问题。以往研究使用线性探测定位任务表征，报告在特定层达到100%分类准确率。我们揭示了一个显著分离：探测准确率完全无法预测因果重要性。对Llama-3.2-3B-despite所有28层进行单位置激活干预，任务转移率为0%，尽管这些位置的探测准确率仍为100%。这一零结果本身即为关键发现，表明任务编码本质上是分布式的。多位置干预（同时替换所有演示输出标记的激活）在第8层实现最高96%的任务转移（N=50，95%置信区间：[87%，99%]），首次精确定位了ICL任务身份的因果位置。我们在四个跨越三种架构家族（LLaMA、Qwen、Gemma）的模型中验证了这一发现，发现普遍存在约30%网络深度的干预窗口。因果追踪揭示不对称结构：查询位置严格必要（破坏度53%-100%），而任意单个演示位置均非必需（破坏度0%），解决了前期研究的关键歧义。关键在于，任务转移依赖内部表征兼容性而非表面相似性（相关系数r=-0.05 vs r=0.31），排除了简单解释。这些结果确立了分布式模板假说：ICL任务身份以分布于演示标记间的输出格式模板形式编码，从根本上重塑了我们对上下文学习机制的理解。
原文摘要 · Abstract (English)
Understanding how large language models encode task identity from few-shot demonstrations is a central open problem in mechanistic interpretability. Prior work uses linear probing to localize task representations, reporting high classification accuracy at specific layers. We reveal a striking dissociation: probing accuracy completely fails to predict causal importance. Single-position activation intervention achieves 0% task transfer across all 28 layers of Llama-3.2-3B-despite 100% probing accuracy at those same positions. This null result is itself a key finding, demonstrating that task encoding is fundamentally distributed. Multi-position intervention-replacing activations at all demonstration output tokens simultaneously-achieves up to 96% transfer (N=50, 95% CI: [87%, 99%]) at layer 8, pinpointing for the first time the causal locus of ICL task identity. We establish the generality of these findings across four models spanning three architecture families (LLaMA, Qwen, Gemma), discovering a universal intervention window at ~30% network depth. Causal tracing uncovers an asymmetric architecture: the query position is strictly necessary (53-100% disruption) while no individual demonstration position is necessary (0% disruption)-resolving a key ambiguity in prior accounts. Crucially, transfer depends on internal representation compatibility, not surface similarity (r=-0.05 vs r=0.31), ruling out trivial explanations. These results establish the distributed template hypothesis: ICL task identity is encoded as output format templates distributed across demonstration tokens, fundamentally reshaping our understanding of how in-context learning operates.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。