测试大模型理解复杂表格层级结构的能力,发现无需专门训练也能胜任。
Hierarchical structure understanding in complex tables with VLLMs: a benchmark and experiments
- 用提示工程探测视觉大模型对表格层级的感知能力
- 在PubTables-1M数据集上构建了复杂层级表基准集CHiTab
- 对比人类与模型表现,揭示通用大模型处理结构化数据潜力
本研究探讨视觉大语言模型(VLLMs)理解科学论文中表格结构的能力。重点考察VLLMs是否能在不进行额外处理的情况下推断表格的层级结构。基于PubTables-1M大规模科学表格语料库,我们提取并构建了复杂层级表(CHiTab)基准数据集,包含具有层级标题的复杂表格。通过一系列提示工程策略,测试多种提示格式与写作风格下模型的理解能力。评估了多个先进的开源权重VLLM,在其默认版本上进行测试,并对部分模型在任务上进行微调。同时在小规模样本上测量人类表现,与模型结果对比。实验表明,未经专门设计用于表格结构理解的通用VLLMs仍具备良好表现。本研究揭示了VLLMs在处理复杂表格方面的潜力与局限,为未来将结构化数据理解融入通用大模型提供指导。
原文摘要 · Abstract (English)
This work investigates the ability of Vision Large Language Models (VLLMs) to understand and interpret the structure of tables in scientific articles. Specifically, we explore whether VLLMs can infer the hierarchical structure of tables without additional processing. As a basis for our experiments we use the PubTables-1M dataset, a large-scale corpus of scientific tables. From this dataset, we extract a subset of tables that we introduce as Complex Hierarchical Tables (CHiTab): a benchmark collection of complex tables containing hierarchical headings. We adopt a series of prompt engineering strategies to probe the models' comprehension capabilities, experimenting with various prompt formats and writing styles. Multiple state-of-the-art open-weights VLLMs are evaluated on the benchmark first using their off-the-shelf versions and then fine-tuning some models on our task. We also measure the performance of humans to solve the task on a small set of tables comparing with performance of the evaluated VLLMs. The experiments support our intuition that generic VLLMs, not explicitly designed for understanding the structure of tables, can perform this task. This study provides insights into the potential and limitations of VLLMs to process complex tables and offers guidance for future work on integrating structured data understanding into general-purpose VLLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。