构建首个真实复杂的分层表格基准，评估大模型处理复杂表格能力
RealHiTBench: A Comprehensive Realistic Hierarchical Table Benchmark for Evaluating LLM-Based Table Analysis
- 设计涵盖多种格式的复杂分层表格数据集
- 25个主流大模型在该基准上表现普遍不佳，验证其挑战性
- 提出树状结构解析方法，提升模型对表格层级的理解
随着大语言模型（LLMs）的快速发展，亟需更具挑战性的基准来评估其处理复杂表格数据的能力。然而，现有基准或基于过时的数据设置，或仅关注简单的扁平表格结构。本文提出RealHiTBench，一个全面的基准，用于评估LLMs和多模态大语言模型（MLLMs）在多种输入格式下处理复杂表格数据的表现，包括LaTeX、HTML和PNG。该基准包含多样化的复杂结构表格，覆盖广泛的任务类型。通过在25个先进LLMs上的实验，结果表明RealHiTBench具有显著挑战性。此外，我们还提出了TreeThinker，一种基于树结构的解析管道，将层级表头组织为树形结构，以增强表格推理能力，验证了改进模型对表格层级感知的重要性。代码与数据已开源。
原文摘要 · Abstract (English)
With the rapid advancement of Large Language Models (LLMs), there is an increasing need for challenging benchmarks to evaluate their capabilities in handling complex tabular data. However, existing benchmarks are either based on outdated data setups or focus solely on simple, flat table structures. In this paper, we introduce RealHiTBench, a comprehensive benchmark designed to evaluate the performance of both LLMs and Multimodal LLMs (MLLMs) across a variety of input formats for complex tabular data, including LaTeX, HTML, and PNG. RealHiTBench also includes a diverse collection of tables with intricate structures, spanning a wide range of task types. Our experimental results, using 25 state-of-the-art LLMs, demonstrate that RealHiTBench is indeed a challenging benchmark. Moreover, we also develop TreeThinker, a tree-based pipeline that organizes hierarchical headers into a tree structure for enhanced tabular reasoning, validating the importance of improving LLMs' perception of table hierarchies. We hope that our work will inspire further research on tabular data reasoning and the development of more robust models. The code and data are available at https://github.com/cspzyy/RealHiTBench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。