让大模型学会按规则一步步算数学题，提升逻辑推理能力。
MetaRuleGPT: Recursive Numerical Reasoning of Language Models Trained with Simple Rules
- 用简单规则训练模型，学习如何组合规则解题。
- 在复杂数学题上准确率显著提升，能分解问题逐步求解。
- 适合想提升模型数学推理能力的研究者或开发者。
近期研究指出大语言模型在数学推理方面存在局限，尤其难以捕捉底层逻辑。受元学习启发，我们提出模型不仅应掌握特定任务知识，还应具备可迁移的问题解决能力。为此，我们提出MetaRuleGPT，一种基于Transformer的新架构，通过学习和组合不同规则，实现精确数值计算与复杂逻辑运算。与传统依赖大量原始实例数据的训练方式不同，MetaRuleGPT在包含基础、复合及迭代规则的抽象数据集上进行预训练。大量实验表明，MetaRuleGPT能模拟人类遵循规则的能力，有效拆解复杂问题并迭代得出准确结果，证明了规则学习在提升语言模型数值推理能力方面的潜力。
原文摘要 · Abstract (English)
Recent studies have highlighted the limitations of large language models in mathematical reasoning, particularly their inability to capture the underlying logic. Inspired by meta-learning, we propose that models should acquire not only task-specific knowledge but also transferable problem-solving skills. We introduce MetaRuleGPT, a novel Transformer-based architecture that performs precise numerical calculations and complex logical operations by learning and combining different rules. In contrast with traditional training sets, which are heavily composed of massive raw instance data, MetaRuleGPT is pre-trained on much less abstract datasets containing basic, compound, and iterative rules for mathematical reasoning. Extensive experimental results demonstrate MetaRuleGPT can mimic human's rule-following capabilities, break down complexity, and iteratively derive accurate results for complex mathematical problems. These findings prove the potential of rule learning to enhance the numerical reasoning abilities of language models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。