arXiv:2501.15581cs.CL2025-01EMNLP被引 12

提出动态误差分类框架,提升大模型数学题推理错误分析与纠正能力

Error Classification of Large Language Models on Math Word Problems: A Dynamically Adaptive Framework

  • 构建动态自适应分类框架,自动识别数学推理中的错误模式
  • 基于30万+错误样本,发现模型能力越强,错误越复杂
  • 引入显式纠错提示,显著提升大模型数学推理表现

大语言模型在多个领域展现强大能力,数学应用题是评估其推理能力的关键基准。现有研究多关注准确率提升,却忽视对错误背后模式的深入理解。当前错误分类依赖静态预定义类别,难以覆盖数学推理中的多样错误模式。为此,我们通过多种采样策略,从15种不同规模的LLM在4个不同数学应用题数据集上收集错误样本,构建了包含304,865个样本的MWPES-300K综合数据集,涵盖广泛错误模式与推理路径。为减少人为偏差并实现细粒度分析,我们提出一种自动化动态误差分类框架。实验表明,数据集特性显著影响错误模式,随着模型能力增强,错误类型由基础向复杂演化。基于此洞察,我们提出错误感知提示(EAP),将常见错误模式作为显式引导,显著提升数学推理性能。

原文摘要 · Abstract (English)

Large Language Models (LLMs) have demonstrated remarkable capabilities across various domains. Math Word Problems (MWPs) serve as a crucial benchmark for evaluating LLMs' reasoning abilities. While most research primarily focuses on improving accuracy, it often neglects understanding and addressing the underlying patterns of errors. Current error classification methods rely on static and predefined categories, which limit their ability to capture the full spectrum of error patterns in mathematical reasoning. To enable systematic error analysis, we collect error samples from 15 different LLMs of varying sizes across four distinct MWP datasets using multiple sampling strategies. Based on this extensive collection, we introduce MWPES-300K, a comprehensive dataset containing 304,865 error samples that cover diverse error patterns and reasoning paths. To reduce human bias and enable fine-grained analysis of error patterns, we propose a novel framework for automated dynamic error classification in mathematical reasoning. Experimental results demonstrate that dataset characteristics significantly shape error patterns, which evolve from basic to complex manifestations as model capabilities increase. With deeper insights into error patterns, we propose Error-Aware Prompting (EAP) that incorporates common error patterns as explicit guidance, leading to significant improvements in mathematical reasoning performance.

大模型数学推理错误分析提示工程

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。