分析表格ICL模型各层贡献，发现部分层可压缩且具可解释性。
Towards Understanding Layer Contributions in Tabular In-Context Learning Models
- 用'层如画师'视角观察表格式上下文学习模型的表征演化
- 仅部分层共享统一表征语言，表明存在结构冗余
- 结果有助于模型压缩与可解释性提升，适合模型优化研究者
尽管表格上下文学习（Tabular ICL）模型与大语言模型（LLMs）在架构上相似，但对其各层如何贡献于表格预测仍知之甚少。本文从'层如画师'视角出发，研究了表格式ICL模型中潜在表征空间随层的变化，识别出可能冗余的层，并与LLMs中的动态进行对比。通过对TabPFN和TabICL的分析发现，仅有部分层共享一致的表征语言，暗示模型存在结构性冗余，为模型压缩与可解释性改进提供了契机。
原文摘要 · Abstract (English)
Despite the architectural similarities between tabular in-context learning (ICL) models and large language models (LLMs), little is known about how individual layers contribute to tabular prediction. In this paper, we investigate how the latent spaces evolve across layers in tabular ICL models, identify potential redundant layers, and compare these dynamics with those observed in LLMs. We analyze TabPFN and TabICL through the "layers as painters" perspective, finding that only subsets of layers share a common representational language, suggesting structural redundancy and offering opportunities for model compression and improved interpretability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。