研究序列相关性如何影响模型上下文学习能力，发现相关性会改变有效上下文长度和注意力架构适配性。
Sequential Correlations Change In-Context Learning: Effective Context Length and Architectural Mismatch
- 用线性注意力构建可解模型，分析序列相关性对上下文学习的影响
- 相关性使有效上下文长度缩短，测试误差在特定注意力下降低
- 揭示了真实序列数据中注意力机制与任务匹配的重要性
现代序列模型具备强大的上下文学习能力，仅凭提示中的示例即可执行新任务。理解这种能力的涌现需要能捕捉自然数据关键特性的理论。线性回归作为ICL理论的有用试验场，但现有研究多集中于独立示例的提示。本文将该设定扩展至具有序列相关性的数据，这是真实序列的基本特征。我们基于线性注意力提出一个可解模型，并在真实的Transformer架构上验证预测。发现两个显著效应：当查询词与上下文无关时，上下文内部相关性导致有效上下文长度缩短，即相关提示的表现等同于更短的独立同分布提示；当查询也与上下文相关时，测试误差下降，尤其在Softmax注意力中相比线性注意力表现更优。结果表明，相关提示不仅改变上下文学习的有效样本量，还影响哪种注意力架构更适合特定任务。
原文摘要 · Abstract (English)
Modern sequence models have a striking capacity for in-context learning (ICL); they can perform new tasks based only on examples given in the prompt. Understanding how this ability emerges requires theory that captures important properties of natural data. Linear regression has served as a useful sandbox for ICL theory, but existing work has largely focused on prompts with independent examples. In this work, we extend this setting to sequentially correlated data, a basic feature of real sequences. We present a solvable model based on linear attention and test our predictions on realistic transformer architectures. We identify two distinct effects: First, when the query token is independent of the context, within-context correlations induce an effective context length: correlated prompts behave like shorter i.i.d. prompts. Second, when the query is also correlated with its context, test error is reduced, particularly for softmax attention when compared to linear attention. These results suggest that correlated prompts alter not only the effective sample size of in-context learning, but also which attention architectures are best matched to the task.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。