构建数学题数据集，测试大模型在无解题时的胡说能力。
TreeCut: A Synthetic Unanswerable Math Word Problem Dataset for LLM Hallucination Evaluation
- 通过树结构拆解题目，系统性生成有解与无解题对。
- GPT-4o在零样本下幻觉率高达64%，o3-mini为44%。
- 题干越复杂、关键条件越居中，越易诱发模型胡说。
大型语言模型（LLMs）在标准数学应用题基准（如GSM8K）上已接近人类表现，但其真实推理能力仍存争议。一个核心问题是，模型常对无解问题给出自信却无根据的答案。我们提出TreeCut，一种合成数据集，通过将每个问题表示为树结构并移除选定的关键条件，系统生成无限数量的无解题及其可解对应题。实验表明，TreeCut能有效诱导大模型（包括GPT-4o和o3-mini）产生幻觉，在零样本设置下最坏情况下的幻觉率分别为64%和44%。进一步分析显示，树结构越深或越复杂、复合项名称越多、移除路径中部的关键条件，均会提高幻觉概率，凸显了大模型识别无解数学问题的持续挑战。数据集生成代码和样例数据已开源：https://github.com/j-bagel/treecut-math。
原文摘要 · Abstract (English)
Large language models (LLMs) now achieve near-human performance on standard math word problem benchmarks (e.g., GSM8K), yet their true reasoning ability remains disputed. A key concern is that models often produce confident, yet unfounded, answers to unanswerable problems. We introduce TreeCut, a synthetic dataset that systematically generates infinite unanswerable math word problems and their answerable counterparts, by representing each question as a tree and removing chosen necessary conditions. Experiments show TreeCut effectively induce hallucinations in large language models, including GPT-4o and o3-mini, with rates of 64% and 44% in their respective worst-case scenarios under zero-shot setting. Further analysis highlights that deeper or more complex trees, composite item names, and removing necessary condition near the middle of a path all increase the likelihood of hallucinations, underscoring the persistent challenges LLMs face in identifying unanswerable math problems. The dataset generation code and sample data are available at https://github.com/j-bagel/treecut-math.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。