揭秘大模型如何理解表格数据，发现注意力机制的三阶段演化规律。
A Closer Look into LLMs for Table Understanding
- 通过分析16个模型，发现注意力从全局扫描到局部聚焦再到强化贡献的三阶段模式。
- 表格任务需更深网络层才能稳定预测，比数学推理更依赖深层特征。
- MoE模型在中间层激活专属表格专家，提示工程可增强对表格的关注度。
尽管大型语言模型（LLMs）在表格理解任务中表现优异，但其内部机制仍不清晰。本文对16种模型（包括通用LLM、专用表格模型及混合专家模型）进行实证研究，探索其处理表格数据与下游任务的机制。重点分析注意力动态、有效层深、专家激活及输入设计影响。关键发现：(1) 模型遵循三阶段注意力模式——早期层广泛扫描表格，中期层定位相关单元格，晚期层强化其贡献；(2) 表格任务需比数学推理更深的层数才能实现稳定预测；(3) MoE模型在中层激活表格式专家，早期与晚期层共享通用专家；(4) 思维链提示可提升表格注意力，经表格微调后进一步增强。这些发现有助于推动表格任务的可解释性与未来研究。
原文摘要 · Abstract (English)
Despite the success of Large Language Models (LLMs) in table understanding, their internal mechanisms remain unclear. In this paper, we conduct an empirical study on 16 LLMs, covering general LLMs, specialist tabular LLMs, and Mixture-of-Experts (MoE) models, to explore how LLMs understand tabular data and perform downstream tasks. Our analysis focus on 4 dimensions including the attention dynamics, the effective layer depth, the expert activation, and the impacts of input designs. Key findings include: (1) LLMs follow a three-phase attention pattern -- early layers scan the table broadly, middle layers localize relevant cells, and late layers amplify their contributions; (2) tabular tasks require deeper layers than math reasoning to reach stable predictions; (3) MoE models activate table-specific experts in middle layers, with early and late layers sharing general-purpose experts; (4) Chain-of-Thought prompting increases table attention, further enhanced by table-tuning. We hope these findings and insights can facilitate interpretability and future research on table-related tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。