发现大模型少样本学习主要依赖功能向量头，而非归纳头。
Which Attention Heads Matter for In-Context Learning?
- 通过消融实验对比两类注意力头的作用。
- 大模型中功能向量头对少样本学习贡献更大。
- 归纳头可能先于功能向量头出现，辅助其形成。
大型语言模型（LLMs）具备出色的上下文学习（ICL）能力，仅通过提示中的少量示例即可完成新任务。已有两种机制解释ICL：归纳头通过查找并复制相关词元，功能向量（FV）头则通过激活计算任务的潜在编码。本文在12个语言模型中研究并比较了这两类头。详细消融实验表明，少样本ICL性能主要依赖于FV头，尤其在大模型中更为显著。此外，发现FV头与归纳头存在关联：许多FV头在训练初期表现为归纳头，随后转变为FV机制。这暗示归纳头可能促进更复杂的FV机制的学习，从而最终驱动ICL。
原文摘要 · Abstract (English)
Large language models (LLMs) exhibit impressive in-context learning (ICL) capability, enabling them to perform new tasks using only a few demonstrations in the prompt. Two different mechanisms have been proposed to explain ICL: induction heads that find and copy relevant tokens, and function vector (FV) heads whose activations compute a latent encoding of the ICL task. To better understand which of the two distinct mechanisms drives ICL, we study and compare induction heads and FV heads in 12 language models. Through detailed ablations, we discover that few-shot ICL performance depends primarily on FV heads, especially in larger models. In addition, we uncover that FV and induction heads are connected: many FV heads start as induction heads during training before transitioning to the FV mechanism. This leads us to speculate that induction facilitates learning the more complex FV mechanism that ultimately drives ICL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。