探究模型架构与任务复杂度对上下文学习的影响
Probing In-Context Learning: Impact of Task Complexity and Model Architecture on Generalization and Efficiency
- 设计复杂任务测试不同模型的上下文学习能力
- Mamba在时序动态任务中表现最优，标准Transformer泛化性强
- 适合研究大模型推理机制与高效训练策略的学者
我们通过系统性实验框架，研究上下文学习（ICL）在任务复杂度和模型架构变化下的表现。除线性回归外，引入高斯核回归与非线性动力系统任务，强调时间与递归推理能力。评估四种模型：GPT2风格Transformer、带FlashAttention的Transformer、基于卷积的Hyena模型及Mamba状态空间模型。所有模型均在合成数据集上从零训练，并在测试阶段评估泛化性能。结果表明，模型架构显著影响ICL效果：标准Transformer在多任务中表现稳健；Mamba在时序结构任务中优势明显；Hyena有效捕捉长程依赖但早期训练方差大；FlashAttention提升计算效率但在低数据下更敏感。进一步分析发现：高斯核任务存在局部性捷径；输入范围缩放增强非线性可分性；课程学习对掌握高维任务至关重要。
原文摘要 · Abstract (English)
We investigate in-context learning (ICL) through a meticulous experimental framework that systematically varies task complexity and model architecture. Extending beyond the linear regression baseline, we introduce Gaussian kernel regression and nonlinear dynamical system tasks, which emphasize temporal and recursive reasoning. We evaluate four distinct models: a GPT2-style Transformer, a Transformer with FlashAttention mechanism, a convolutional Hyena-based model, and the Mamba state-space model. Each model is trained from scratch on synthetic datasets and assessed for generalization during testing. Our findings highlight that model architecture significantly shapes ICL performance. The standard Transformer demonstrates robust performance across diverse tasks, while Mamba excels in temporally structured dynamics. Hyena effectively captures long-range dependencies but shows higher variance early in training, and FlashAttention offers computational efficiency but is more sensitive in low-data regimes. Further analysis uncovers locality-induced shortcuts in Gaussian kernel tasks, enhanced nonlinear separability through input range scaling, and the critical role of curriculum learning in mastering high-dimensional tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。