arXiv:2412.16964cs.AIcs.CL2024-12被引 5

用人类与AI协作生成细粒度推理轨迹,提升大模型数学解题能力。

System-2 Mathematical Reasoning via Enriched Instruction Tuning

  • 通过人类标注+AI反馈生成分步推理链,增强训练数据质量。
  • 在GSM8K上达到84.1%准确率,超越多数纯提示与微调方法。
  • 无需符号验证工具,适合追求高可信数学推理的开发者使用。

通过系统2式推理解决复杂数学问题本是人类自然能力,但对当前大语言模型仍是重大挑战。我们识别出缺乏刻意多步推理数据为主要瓶颈。为此提出丰富指令微调(EIT),通过融合人类与AI反馈,扩充已有数学标注数据集,生成细粒度推理轨迹。这些数据用于微调开源LLM,显著提升其数学推理能力,且无需依赖任何符号验证程序。EIT包含两个关键步骤:用推理计划(ERP)生成高层分解目标序列;用推理步骤(ERS)补充人类标注常忽略的推理上下文,形成更连贯的推理路径。与仅依赖模型内部知识生成思维链的方法不同,EIT以人类标注初始答案作为‘元知识’,引导模型生成更详细精准的推理过程,从而构建更可信的数学专家模型。实验表明,EIT在GSM8K上达84.1%准确率,在MATH上达32.5%,超越现有最先进微调与提示方法,甚至媲美工具增强方法。

原文摘要 · Abstract (English)

Solving complex mathematical problems via system-2 reasoning is a natural human skill, yet it remains a significant challenge for current large language models (LLMs). We identify the scarcity of deliberate multi-step reasoning data as a primary limiting factor. To this end, we introduce Enriched Instruction Tuning (EIT), a method that enriches existing human-annotated mathematical datasets by synergizing human and AI feedback to create fine-grained reasoning trajectories. These datasets are then used to fine-tune open-source LLMs, enhancing their mathematical reasoning abilities without reliance on any symbolic verification program. Concretely, EIT is composed of two critical steps: Enriching with Reasoning Plan (ERP) and Enriching with Reasoning Step (ERS). The former generates a high-level plan that breaks down complex instructions into a sequence of simpler objectives, while ERS fills in reasoning contexts often overlooked by human annotators, creating a smoother reasoning trajectory for LLM fine-tuning. Unlike existing CoT prompting methods that generate reasoning chains only depending on LLM's internal knowledge, our method leverages human-annotated initial answers as ``meta-knowledge'' to help LLMs generate more detailed and precise reasoning processes, leading to a more trustworthy LLM expert for complex mathematical problems. In experiments, EIT achieves an accuracy of 84.1% on GSM8K and 32.5% on MATH, surpassing state-of-the-art fine-tuning and prompting methods, and even matching the performance of tool-augmented methods.

数学推理指令微调推理链大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。