通过动态提前停止提升表格上下文学习推理速度
Early Stopping Tabular In-Context Learning
- 每层Transformer后动态判断是否停止上下文学习
- 小任务提速1.3倍，大任务提速2.2倍，性能损失极小
- 适合追求高效推理的表格学习应用
表格基础模型在各类表格学习任务中通过上下文学习表现出色，无需微调即可实现强泛化能力。然而，其推理成本仍较高，尤其在大数据集上。为此，本文提出在上下文学习过程中引入早期停止机制：在每个Transformer编码器层后动态评估是否终止学习过程。一旦停止，使用预训练的分层解码器生成嵌入表示。在34个小型分类任务上的实验表明，该方法可使推理速度提升最多1.3倍，预测性能几乎无损。进一步在5个较大分类任务上评估，最大提速达2.2倍。结果表明，早期退出是提升表格上下文学习效率的有效且实用策略。
原文摘要 · Abstract (English)
Tabular foundation models have shown strong performance across various tabular learning tasks via in-context learning, offering robust generalization without any downstream finetuning. However, their inference-time costs remain high, particularly for larger datasets. To address this, we propose early-stopping the in-context learning process. We achieve this by dynamically evaluating whether to stop in-context learning after each Transformer encoder layer. Once stopped, we decode the embedding using a pre-trained layer-wise decoder. Experiments across 34 small classification tasks size show that early stopping in-context learning accelerates inference by up to x1.3 with negligible degradation in predictive performance. To assess scalability, we further evaluate our method on five larger classification tasks, achieving speedups of up to x2.2. Our results demonstrate the potential of early exiting as an effective and practical strategy for improving the efficiency of tabular in-context learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。