研究动态环境下Transformer如何通过门控注意力自适应学习。
Learning to Adapt: In-Context Learning Beyond Stationarity
- 提出门控线性注意力机制，实现对变化任务的自适应响应。
- 理论证明其在非平稳回归中比普通注意力误差更低。
- 适合关注动态场景下少样本学习的研究者。
Transformer模型因其出色的实证性能成为科学与工程领域的基础架构。其成功关键在于上下文学习（ICL）：在未见过的任务提示下，无需参数更新即可进行逐标记和下一标记预测。近期理论研究开始揭示其背后的机制，尤其在监督回归设置中。然而，这些分析大多假设任务分布平稳，忽略了现实中目标函数随时间变化的广泛场景。本文填补这一空白，首次对非平稳回归问题下的ICL提供理论分析。研究发现，门控线性注意力（GLA）能有效适应不断变化的输入-输出关系，并在理论上证明其优于标准线性注意力。采用一阶自回归过程建模非平稳性，结果表明：GLA通过动态调节历史输入影响，实现可学习的时效偏好，显著降低训练与测试误差。实验进一步验证了门控机制在非平稳ICL任务中的优势。
原文摘要 · Abstract (English)
Transformer models have become foundational across a wide range of scientific and engineering domains due to their strong empirical performance. A key capability underlying their success is in-context learning (ICL): when presented with a short prompt from an unseen task, transformers can perform per-token and next-token predictions without any parameter updates. Recent theoretical efforts have begun to uncover the mechanisms behind this phenomenon, particularly in supervised regression settings. However, these analyses predominantly assume stationary task distributions, which overlook a broad class of real-world scenarios where the target function varies over time. In this work, we bridge this gap by providing a theoretical analysis of ICL under non-stationary regression problems. We study how the gated linear attention (GLA) mechanism adapts to evolving input-output relationships and rigorously characterize its advantages over standard linear attention in this dynamic setting. To model non-stationarity, we adopt a first-order autoregressive process and show that GLA achieves lower training and testing errors by adaptively modulating the influence of past inputs -- effectively implementing a learnable recency bias. Our theoretical findings are further supported by empirical results, which validate the benefits of gating mechanisms in non-stationary ICL tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。