揭示大模型ICL的两种分离机制：识别任务类型与绑定输入输出。
Task Schema and Binding: A Double Dissociation Study of In-Context Learning
- 通过激活修补实验，发现任务模式和绑定关系可独立传递
- 任务模式转移率100%，绑定转移率62%，且受先验知识干扰
- 适用于各种架构，尤其解释了为何某些提示会失效
我们通过跨9个模型（7个Transformer家族及Mamba，参数量370M-13B）的激活修补实验，因果验证了上下文学习（ICL）可分解为两个可分离机制：任务模式（抽象任务类型识别）与绑定（特定输入输出关联）。关键发现包括：1. 双分离现象：通过后期MLP修补，任务模式实现100%转移；通过残差流修补，绑定转移率为62%，证明二者神经可分；2. 先验-模式权衡：模式依赖度与先验知识呈负相关（斯皮尔曼ρ = -0.596, p < 0.001，N=28任务-模型对）；3. 架构普适性：该机制在所有测试架构中均成立，包括非Transformer的Mamba。结果表明，当缺乏先验知识时，模型完全依赖任务模式；而先验知识通过注意力错位（72.7%近期偏倚）而非输出竞争（0%）造成干扰。这解释了为何任意映射有效（无先验时全靠模式），但事实覆盖失败——瓶颈在于注意力，而非输出层。实际意义：理解双机制可优化提示工程——利用稳定模式转移减少演示需求，结合先验设计可缓解高先验场景下38%的绑定失败率，提升生产系统可靠性。
原文摘要 · Abstract (English)
We provide causal mechanistic validation that in-context learning (ICL) decomposes into two separable mechanisms: Task Schema (abstract task type recognition) and Binding (specific input-output associations). Through activation patching experiments across 9 models from 7 Transformer families plus Mamba (370M-13B parameters), we establish three key findings: 1. Double dissociation: Task Schema transfers at 100% via late MLP patching; Binding transfers at 62% via residual stream patching -- proving separable mechanisms 2. Prior-Schema trade-off: Schema reliance inversely correlates with prior knowledge (Spearman rho = -0.596, p < 0.001, N=28 task-model pairs) 3. Architecture generality: The mechanism operates across all tested architectures including the non-Transformer Mamba These findings offer a mechanistic account of the ICL puzzle that contrasts with prior views treating ICL as a monolithic mechanism (whether retrieval-based, gradient descent-like, or purely Bayesian). By establishing that Schema and Binding are neurally dissociable -- not merely behavioral modes -- we provide causal evidence for dual-process theories of ICL. Models rely on Task Schema when prior knowledge is absent, but prior knowledge interferes through attentional mis-routing (72.7% recency bias) rather than direct output competition (0%). This explains why arbitrary mappings succeed (zero prior leads to full Schema reliance) while factual overrides fail -- and reveals that the true bottleneck is attentional, not output-level. Practical implications: Understanding these dual mechanisms enables more efficient prompt engineering -- reliable schema transfer reduces required demonstrations for novel tasks, while prior-aware design can mitigate the 38% binding failure rate in high-prior scenarios, improving ICL system reliability in production deployments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。