测试大模型对数学文本的逻辑推理能力，发现投票机制可接近人工标注效果。
Math Natural Language Inference: this should be easy!
- 构建数学领域NLI数据集，前提来自真实数学文献，假设由专家标注
- 多模型投票在部分场景下表现接近人工标注，但模型仍难处理基础数学推理
- 提供开源数据集，助力未来数学语言理解研究
我们探究当代大语言模型是否具备对数学文本进行自然语言推理（NLI）的能力，提出“数学NLI”问题。构建了一个数学NLI语料库，其前提来自已有数学文献，假设与真实标签由兼具科研级数学背景和NLI经验的专家提供。同时考察使用模型自身生成假设的数据集质量。不仅评估性能，还分析了不同模型组间的推理一致性。研究发现：在某些情况下，多个模型的多数投票结果近似于人工标注水平；但模型在数学语言理解上仍有明显短板，常无法完成基本推理。相较于前代模型，当前模型较少出现仅基于假设的“虚假推理”。此外，我们公开了该数据集以支持后续数学语言推理研究。
原文摘要 · Abstract (English)
We ask whether contemporary LLMs are able to perform natural language inference (NLI) tasks on mathematical texts. We call this the Math NLI problem. We construct a corpus of Math NLI pairs whose premises are from extant mathematical text and whose hypotheses and gold labels were provided by people with experience in both research-level mathematics and also in the NLI field. We also investigate the quality of corpora using the same premises but whose hypotheses are provided by LLMs themselves. We not only investigate the performance but also the inter-group consistency of the diverse group of LLMs. We have both positive and negative findings. Among our positive findings: in some settings, using a majority vote of LLMs is approximately equivalent to using human-labeled data in the Math NLI area. On the negative side: LLMs still struggle with mathematical language. They occasionally fail at even basic inferences. Current models are not as prone to hypothesis-only "inference" in our data the way the previous generation had been. In addition to our findings, we also provide our corpora as data to support future work on Math NLI.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。