揭示上下文如何通过权重微调影响Transformer模型
Equivalence of Context and Parameter Updates in Modern Transformer Blocks
- 上下文信息可转化为MLP权重的低秩修正
- 理论证明适用于Gemma等主流LLM架构
- 为理解提示如何改写模型行为提供新视角
近期研究发现，原始Transformer中上下文的影响可通过构建与标记相关的秩-1修正项隐式地作用于MLP权重。本文将这一基础理论扩展至现代大语言模型的多样化架构。首先，针对Gemma风格的Transformer块，我们给出了精确的解析解，证明上下文效果可完全映射为对MLP权重矩阵的秩-1修正以及对RMSNorm尺度的修正。随后，我们进一步提出构造性证明与算法，适用于多层模型。为统一这些发现，引入以输入可控性和输出可控性为核心的通用框架，并证明：只要内层函数具备输入可控性、外层函数具备输出可控性，任意MLP模块均可实现完美的隐式权重修正。该框架适用于广泛的现代LLM架构，包括门控机制、预/后归一化、专家混合（MoE）及串行/并行Transformer块。
原文摘要 · Abstract (English)
Recent research has established that the impact of context in a vanilla transformer can be represented implicitly by forming a token-dependent, rank-1 patch to its MLP weights. This work extends that foundational theory to the diverse architectures of modern Large Language Models. We first demonstrate a precise, analytical solution for a Gemma-style transformer block, proving that the entire effect of a context can be perfectly mapped to rank-1 patches on its MLP weight matrices and a patch to the RMSNorm scale. We then generalize this result, providing a constructive proof and algorithm for multi-layer models. To unify these findings, we introduce a general framework centered on two core properties: input controllability and output controllability. We prove that a perfect implicit weight patch is possible for any MLP block where the inner function is input-controllable and the outer function is output-controllable. This provides a simpler and more powerful lens for understanding how transformer models transmute prompts into effective weights. This setup generalizes to a wide range of modern LLM architectures including gating, pre-/post-norm, mixture of experts and sequential/parallel transformer blocks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。