arXiv:2506.05587cs.AIcs.CL2025-06NeurIPS被引 22

构建25类真实表格任务的大型评测基准,检验模型专家级表处理能力。

MMTU: A Massive Multi-Task Table Understanding and Reasoning Benchmark

  • 涵盖28000+问题的跨任务表格理解与推理评测集
  • 顶尖模型如GPT-5仅达69%准确率,显示当前技术仍不足
  • 适合研究结构化数据处理、大模型表格分析的学者和工程师

表格及其应用场景在实际系统中至关重要,如电子表格、数据库和计算笔记本,传统上需数据工程师、分析师等专业人员操作。尽管大语言模型在表格任务(如电子表格助手、数据库助手)中取得显著进展,但相关全面评测仍严重缺乏。现有评测多集中于NL-to-SQL和表问答等有限任务,忽视了专业用户面临的复杂真实场景。为此,本文提出MMTU,一个包含25种真实世界表格任务、超过28,000个问题的大规模基准,旨在全面评估模型对表格的理解、推理与操作能力。这些任务源于数十年来关于表格数据的计算机科学研究,聚焦于专业用户所面对的复杂挑战。我们发现,即使最前沿的模型(如OpenAI GPT-5和DeepSeek R1)在该基准上的得分也仅为约69%和57%,表明当前模型在结合表格理解、推理与编程等综合能力方面仍有巨大提升空间。本工作期望通过此基准推动基础模型在结构化数据处理与分析方面的进一步发展。代码与数据已开源:https://github.com/MMTU-Benchmark/MMTU 及 https://huggingface.co/datasets/MMTU-benchmark/MMTU。

原文摘要 · Abstract (English)

Tables and table-based use cases play a crucial role in many important real-world applications, such as spreadsheets, databases, and computational notebooks, which traditionally require expert-level users like data engineers, data analysts, and database administrators to operate. Although LLMs have shown remarkable progress in working with tables (e.g., in spreadsheet and database copilot scenarios), comprehensive benchmarking of such capabilities remains limited. In contrast to an extensive and growing list of NLP benchmarks, evaluations of table-related tasks are scarce, and narrowly focus on tasks like NL-to-SQL and Table-QA, overlooking the broader spectrum of real-world tasks that professional users face. This gap limits our understanding and model progress in this important area. In this work, we introduce MMTU, a large-scale benchmark with over 28K questions across 25 real-world table tasks, designed to comprehensively evaluate models ability to understand, reason, and manipulate real tables at the expert-level. These tasks are drawn from decades' worth of computer science research on tabular data, with a focus on complex table tasks faced by professional users. We show that MMTU require a combination of skills -- including table understanding, reasoning, and coding -- that remain challenging for today's frontier models, where even frontier reasoning models like OpenAI GPT-5 and DeepSeek R1 score only around 69\% and 57\% respectively, suggesting significant room for improvement. We highlight key findings in our evaluation using MMTU and hope that this benchmark drives further advances in understanding and developing foundation models for structured data processing and analysis. Our code and data are available at https://github.com/MMTU-Benchmark/MMTU and https://huggingface.co/datasets/MMTU-benchmark/MMTU.

表格理解大模型评测结构化数据

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。