构建数学符号理解基准，评估大模型在科技文档中的符号推理能力
STEM-POM: Evaluating Language Models Math-Symbol Reasoning in Document Parsing
- 设计涵盖2000+数学符号的上下文理解评测集
- 顶尖模型分类准确率仅20%-60%，微调后达50%-60%
- 适合关注数学推理与文档理解的研究者
大语言模型在科学、技术、工程和数学（STEM）文档中解析数学符号的能力仍受限。本文提出STEM-PoM，一个基于真实ArXiv文献的综合性基准数据集，包含超过2000个数学符号，按主属性分为变量、常数、运算符和单位描述符，并细分为标量/向量/矩阵等子属性，以及局部/全局/学科特定等标签。实验表明，当前最先进的大模型在上下文学习下平均准确率为20%-60%，微调后可达50%-60%，显示其对数学符号理解存在显著不足。提升符号分类能力有助于增强模型在下游数学推理任务中的表现。代码与数据已开源。
原文摘要 · Abstract (English)
Advances in large language models (LLMs) have spurred research into enhancing their reasoning capabilities, particularly in math-rich STEM (Science, Technology, Engineering, and Mathematics) documents. While LLMs can generate equations or solve math-related queries, their ability to fully understand and interpret abstract mathematical symbols in long, math-rich documents remains limited. In this paper, we introduce STEM-PoM, a comprehensive benchmark dataset designed to evaluate LLMs' reasoning abilities on math symbols within contextual scientific text. The dataset, sourced from real-world ArXiv documents, contains over 2K math symbols classified as main attributes of variables, constants, operators, and unit descriptors, with additional sub-attributes including scalar/vector/matrix for variables and local/global/discipline-specific labels for both constants and operators. Our extensive experiments demonstrate that state-of-the-art LLMs achieve an average accuracy of 20-60% under in-context learning and 50-60% with fine-tuning, highlighting a substantial gap in their ability to classify mathematical symbols. By improving LLMs' mathematical symbol classification, STEM-PoM further enhances models' downstream mathematical reasoning capabilities. The code and data are available at https://github.com/jiaruzouu/STEM-PoM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。