TTT让Transformer在测试时微调，显著减少少样本学习所需数据量。
Test-Time Training Provably Improves Transformers as In-context Learners
- 用单步梯度更新模型权重，适配测试时的上下文示例。
- 理论证明可缓解分布偏移，使少样本学习所需样本量减少3至5倍。
- 适合需要高效推理的少样本任务，如表格分类，训练成本几乎忽略不计。
测试时训练（TTT）方法通过显式更新模型权重来适应特定测试实例，在语言建模与推理等场景中表现优异。本文针对基于梯度的TTT算法在上下文学习中的应用进行研究，聚焦于在测试提示中提供的上下文示例上训练Transformer模型。我们对线性Transformer在单步梯度更新下的行为提供全面的理论分析，揭示了预训练分布与目标任务间对齐的作用，阐明了TTT如何缓解分布偏移，并量化了其样本复杂度——表明其可显著降低实现上下文学习所需的最终样本量。作为实证贡献，我们研究了TTT在TabPFN（一种表格基础模型）上的效果。实验结果与理论一致：TTT使表格分类任务所需样本量减少3至5倍，极大提升了推理效率，且训练成本极低。
原文摘要 · Abstract (English)
Test-time training (TTT) methods explicitly update the weights of a model to adapt to the specific test instance, and they have found success in a variety of settings, including most recently language modeling and reasoning. To demystify this success, we investigate a gradient-based TTT algorithm for in-context learning, where we train a transformer model on the in-context demonstrations provided in the test prompt. Specifically, we provide a comprehensive theoretical characterization of linear transformers when the update rule is a single gradient step. Our theory (i) delineates the role of alignment between pretraining distribution and target task, (ii) demystifies how TTT can alleviate distribution shift, and (iii) quantifies the sample complexity of TTT including how it can significantly reduce the eventual sample size required for in-context learning. As our empirical contribution, we study the benefits of TTT for TabPFN, a tabular foundation model. In line with our theory, we demonstrate that TTT significantly reduces the required sample size for tabular classification (3 to 5 times fewer) unlocking substantial inference efficiency with a negligible training cost.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。