评测大模型在嘈杂数学试卷中的结构提取与拒答能力
MathDoc: Benchmarking Structured Extraction and Active Refusal on Noisy Mathematics Exam Papers
- 构建首个真实高中数学试卷的文档级信息抽取基准
- 3609个带噪声题目，含无法识别样本以测试拒答能力
- 发现主流大模型面对模糊输入仍强行输出，缺乏可靠拒答
从纸质数学试卷中自动提取结构化问题对智能教育至关重要，但在真实场景下因严重视觉噪声仍具挑战。现有基准多关注清晰文档或通用版面分析，忽视数学问题的结构完整性及模型对不完整输入的主动拒答能力。本文提出MathDoc，首个针对真实高中数学试卷的文档级信息抽取基准，包含3,609个精心筛选的问题，带有真实世界瑕疵，并明确引入无法识别样本以评估模型的主动拒答行为。我们设计了多维度评估框架，涵盖题干准确率、视觉相似度和拒答能力。在SOTA多模态大模型（如Qwen3-VL和Gemini-2.5-Pro）上的实验表明，尽管端到端模型提取性能较强，但普遍无法拒绝模糊输入，反而生成自信但无效的输出。这一结果揭示了当前多模态大模型在劣质文档条件下的可靠性关键缺陷，并确立MathDoc作为评估模型鲁棒性的基准。
原文摘要 · Abstract (English)
The automated extraction of structured questions from paper-based mathematics exams is fundamental to intelligent education, yet remains challenging in real-world settings due to severe visual noise. Existing benchmarks mainly focus on clean documents or generic layout analysis, overlooking both the structural integrity of mathematical problems and the ability of models to actively reject incomplete inputs. We introduce MathDoc, the first benchmark for document-level information extraction from authentic high school mathematics exam papers. MathDoc contains \textbf{3,609} carefully curated questions with real-world artifacts and explicitly includes unrecognizable samples to evaluate active refusal behavior. We propose a multi-dimensional evaluation framework covering stem accuracy, visual similarity, and refusal capability. Experiments on SOTA MLLMs, including Qwen3-VL and Gemini-2.5-Pro, show that although end-to-end models achieve strong extraction performance, they consistently fail to refuse illegible inputs, instead producing confident but invalid outputs. These results highlight a critical gap in current MLLMs and establish MathDoc as a benchmark for assessing model reliability under degraded document conditions. Our project repository is available at \href{https://github.com/winnk123/papers/tree/master}{GitHub repository}
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。