用编程题生成复杂逻辑推理数据，提升模型推理能力
LogicPro: Improving Complex Logical Reasoning via Program-Guided Learning
- 基于编程题和代码中间变量生成文本推理过程
- 540K合成数据在多个基准上超越现有方法
- 适合需要强逻辑推理的AI模型训练与评估
本文提出一种名为LogicPro的新数据合成方法，利用LeetCode风格的算法题目及其对应的程序解法，以文本形式合成复杂的逻辑推理数据。首先，通过源算法题和测试用例生成复杂推理问题；然后，基于标准Python解法和测试用例获取每道题的标准答案及中间变量输出；最后，在代码中间变量的引导下，为每个推理问题合成文本推理过程。该方法可生成难度高、可扩展、有效且具备标准答案与高质量推理链的数据。仅基于2,360个算法题构建的540K合成数据集，使多个模型在BBH$^{27}$、LogicBench、DROP、AR-LSAT和GSM8K等数据集上取得显著提升，性能优于大量现有推理数据集。
原文摘要 · Abstract (English)
In this paper, we propose a new data synthesis method called \textbf{LogicPro}, which leverages LeetCode-style algorithm \underline{Pro}blems and their corresponding \underline{Pro}gram solutions to synthesize Complex \underline{Logic}al Reasoning data in text format. First, we synthesize complex reasoning problems through source algorithm problems and test cases. Then, standard answers and intermediate variable outputs are obtained for each problem based on standard python solutions and test cases. Finally, with the guidance of code intermediate variables, we synthesize the text reasoning process for each reasoning problems. Through this method, we can synthesize data that is difficult, scalable, effective, and comes with golden standard answers and high-quality reasoning processes. As a result, with our 540K synthesized dataset constructed solely from 2,360 algorithm problems, our approach \footnote{Code and data are publicly available at https://github.com/jiangjin1999/LogicPro} achieves significant improvements in multiple models for the datasets \textit{BBH$^{27}$}, \textit{LogicBench}, \textit{DROP}, \textit{AR-LSAT}, and \textit{GSM8K}, etc. outperforming a wide range of existing reasoning datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。