测试时训练让大模型在任务上更专精，提升性能。
Specialization after Generalization: Towards Understanding Test-Time Training in Foundation Models
- 大模型测试时继续训练，聚焦任务相关概念
- 实验证明测试时训练可显著降低分布内误差
- 适合需要高精度的特定任务场景
近期实证研究探索了在测试阶段继续训练模型（即测试时训练，TTT）的方法，发现其能显著提升性能。然而，对TTT为何有效、何时有效的理解仍有限。早期解释多基于分布外适应或特权数据，但随着基础模型规模增大，多数测试数据为分布内，这些解释面临挑战。本文提出，基础模型在全局仍处于欠参数状态，TTT提供了一种‘通用之后再专精’的机制，使模型容量集中于测试任务相关概念。基于线性表示假设，我们构建模型并证明TTT在分布内测试误差上优于全局训练。通过在ImageNet上训练稀疏自编码器，我们验证了语义相关样本由少数共享概念解释。跨图像与语言任务的缩放实验进一步确认了该模型的实际意义，识别出专精最有效的场景。
原文摘要 · Abstract (English)
Recent empirical studies have explored the idea of continuing to train a model at test-time for a given task, known as test-time training (TTT), and have found it to yield significant performance improvements. However, there is limited understanding of why and when TTT is effective. Earlier explanations mostly focused on the observation that TTT may help when applied to out-of-distribution adaptation or used with privileged data. However, the growing scale of foundation models with most test data being in-distribution questions these explanations. We instead posit that foundation models remain globally underparameterized, with TTT providing a mechanism for specialization after generalization, focusing capacity on concepts relevant to the test task. Specifically, under the linear representation hypothesis, we propose a model in which TTT achieves a substantially smaller in-distribution test error than global training. We empirically validate our model's key assumptions by training a sparse autoencoder on ImageNet, showing that semantically related data points are explained by only a few shared concepts. Finally, we perform scaling studies across image and language tasks that confirm the practical implications of our model, identifying the regimes where specialization is most effective.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。