让表格理解更高效：通过逐步提问引导和聚焦关键信息，减少冗余计算。
TabFlash: Efficient Table Understanding with Progressive Question Conditioning and Token Focusing
- 逐步注入问题信息到视觉模型，生成针对性强的特征。
- 剪枝背景令牌并训练模型聚焦核心内容，性能提升且资源消耗更低。
- 适合需要快速精准解析表格的场景，如金融、科研数据处理。
表格图像因需关注特定问题且存在冗余背景区域，给高效理解带来挑战。现有多模态大模型常忽视这些特性，导致视觉表示冗余无用。为此，我们提出渐进式问题条件化，将问题以递增频率注入视觉变换器各层，利用每层信息承载能力生成问题感知特征；为减少冗余，引入剪枝策略剔除背景令牌以提升效率；为缓解剪枝带来的信息损失，进一步提出令牌聚焦训练策略，促使模型将关键信息集中于保留的令牌中。结合上述方法，我们构建了TabFlash，一种高效且有效的表格理解多模态大模型。该模型在多项指标上达到当前最优表现，优于开源与专有大模型，同时相比第二优模型降低27%的浮点运算量（FLOPs）和30%的内存占用。
原文摘要 · Abstract (English)
Table images present unique challenges for effective and efficient understanding due to the need for question-specific focus and the presence of redundant background regions. Existing Multimodal Large Language Model (MLLM) approaches often overlook these characteristics, resulting in uninformative and redundant visual representations. To address these issues, we aim to generate visual features that are both informative and compact to improve table understanding. We first propose progressive question conditioning, which injects the question into Vision Transformer layers with gradually increasing frequency, considering each layer's capacity to handle additional information, to generate question-aware visual features. To reduce redundancy, we introduce a pruning strategy that discards background tokens, thereby improving efficiency. To mitigate information loss from pruning, we further propose token focusing, a training strategy that encourages the model to concentrate essential information in the retained tokens. By combining these approaches, we present TabFlash, an efficient and effective MLLM for table understanding. TabFlash achieves state-of-the-art performance, outperforming both open-source and proprietary MLLMs, while requiring 27% less FLOPs and 30% less memory usage compared to the second-best MLLM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。