发现表格模型推理可简化为单层，性能不降反升
Is One Layer Enough? Understanding Inference Dynamics in Tabular Foundation Models
- 分析6个先进表格模型的逐层推理过程
- 发现多层间存在大量重复计算，仅用20%参数即可持平性能
- 适合对模型效率敏感的研究者与工业界应用
基于Transformer的表格基础模型（TFMs）在中小规模表格预测任务中占据主导地位，但其推理机制仍不明确。我们首次对6个最先进的表格上下文学习模型进行了大规模机制研究，探索预测在深度方向上的演化过程，识别出不同的推理阶段，并揭示了与语言模型不同的隐空间动态。研究发现多个模型存在显著的深度冗余，表明推理过程中存在迭代优化且计算重叠。基于此，我们设计了一个概念验证的环形单层模型，仅使用原始模型20%的参数，便实现了相当的性能。代码已开源。
原文摘要 · Abstract (English)
Transformer-based tabular foundation models (TFMs) dominate small to medium tabular predictive benchmark tasks, yet their inference mechanisms remain largely unexplored. We present the first large-scale mechanistic study of layerwise dynamics in 6 state-of-the-art tabular in-context learning models. We explore how predictions emerge across depth, identify distinct stages of inference and reveal latent-space dynamics that differ from those of language models. Our findings indicate substantial depthwise redundancy across multiple models, suggesting iterative refinement with overlapping computations during inference stages. Guided by these insights, we design a proof-of-concept, looped single-layer model that uses only 20% of the original model's parameters while achieving comparable performance. The code is available at https://github.com/amirbalef/is_one_layer_enough.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。