用大模型自动生成海量高质量数学题,解决推理能力训练数据不足问题。
Training and Evaluating Language Models with Template-based Data Generation
- 用GPT-4生成可参数化的元模板,自动构造复杂数学题
- 构建超700万条带可验证解的数学题数据集,支持监督微调
- 适合研究模型推理、数据增强或强化学习对齐的学者
大型语言模型(如GPT-3、PaLM、Llama)在自然语言处理中展现出强大能力,但在需要多步推理的复杂任务(如数学求解)上仍表现不佳,主要因缺乏大规模高质量领域数据。为此,本文提出基于模板的数据生成方法(TDG),利用前沿大模型GPT-4自动生成参数化元模板,进而合成近乎无限量的高质量问题与解法。基于此,我们构建了首个超过700万条样本的合成数据集TemplateMath Part I:TemplateGSM,每道题均配有程序可验证的解,极大提升数据质量与规模。该资源不仅缓解了监督微调中的数据稀缺问题,还为基于可验证奖励的强化学习(RLVR)提供了可靠对齐机制。通过引入元模板自动生成,实现问题结构多样化与复杂性,突破数据生成与验证瓶颈,推动具备强而可靠推理能力的新一代大模型发展。
原文摘要 · Abstract (English)
The rapid advancement of large language models (LLMs) such as GPT-3, PaLM, and Llama has significantly transformed natural language processing, showcasing remarkable capabilities in understanding and generating language. However, a fundamental bottleneck persists: these models often struggle with tasks requiring complex, multi-step reasoning, particularly in mathematical problem-solving. This deficiency stems from the critical scarcity of large-scale, high-quality, domain-specific datasets necessary for cultivating sophisticated reasoning abilities. To overcome this challenge, we introduce Template-based Data Generation (TDG), a novel and scalable paradigm that harnesses frontier LLMs (GPT-4) to automatically generate parameterized meta-templates, which in turn synthesize a virtually infinite stream of high-quality problems and solutions. Using this paradigm, we create TemplateMath Part I: TemplateGSM, a foundational dataset of over 7 million synthetically generated grade school math problems. Each problem is accompanied by a programmatically verifiable solution, offering an unprecedented level of quality at scale. This resource not only resolves the data scarcity issue for supervised fine-tuning but also provides a robust mechanism for model alignment through Reinforcement Learning with Verifiable Rewards (RLVR). Our approach elevates data augmentation by leveraging GPT-4 to generate meta-templates, ensuring diverse and complex problem structures. By providing a scalable solution to the data and verification bottleneck, TDG and TemplateGSM pave the way for a new generation of LLMs with powerful, reliable reasoning skills.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。