Transformer通过自注意力与MLP协同，实现推理时隐式更新权重，无需训练即可学习新模式。
Learning without training: The implicit dynamics of in-context learning
- 利用自注意力与MLP的叠加机制，在推理中隐式调整MLP权重。
- 前向传播等价于在无上下文时对MLP做低秩权重更新。
- 为大模型在推理时学习新模式提供了理论解释，适合研究AI机制者阅读。
大型语言模型（LLMs）最引人注目的特性之一是其具备上下文学习能力：在推理阶段，即使这些模式未在训练中出现，仅通过提示中的示例即可学习新规律，且无需任何权重更新。其内在机理仍不清楚。本文通过理论分析和实验表明，标准的Transformer块中自注意力层与MLP层的堆叠结构，可使MLP层的权重在上下文影响下发生隐式调整。具体而言，带上下文的前向传播在数学上等价于不带上下文但对MLP权重进行最小低秩更新的前向传播。这一机制可能解释了为何大模型能在训练之外展现出上下文学习能力。
原文摘要 · Abstract (English)
One of the most striking features of Large Language Models (LLMs) is their ability to learn in-context. Namely at inference time an LLM is able to learn new patterns without any additional weight update when these patterns are presented in the form of examples in the prompt, even if these patterns were not seen during training. The mechanisms through which this can happen are still largely unknown. In this work, we show that the stacking of a self-attention layer with an MLP allows the transformer block to implicitly modify the weights of the MLP layer according to the context. We argue through theoretical analysis and experimentation that this simple mechanism may help explain why LLMs demonstrate capabilities of in-context learning, beyond what is captured during training. Specifically, we show that a standard forward pass with context is mathematically equivalent to a forward pass without context but with the MLP weights updated by a minimal low-rank update representing the context.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。