用强化学习生成高难度数学题,提升大模型推理能力。
MathSmith: Towards Extremely Hard Mathematical Reasoning by Forging Synthetic Problems with a Reinforced Policy
- 从头构造新题目,结合概念对与强化学习优化复杂度。
- 在5个基准上均超越基线,长链推理下表现更优。
- 可针对薄弱知识点定制训练题,适合研究推理模型者。
大型语言模型在数学推理上取得显著进展,但受限于高质量、高难度训练数据的稀缺。现有合成方法多依赖人类编写模板的转换,导致多样性与可扩展性不足。本文提出MathSmith框架,通过从PlanetMath中随机采样概念-解释对,从零构建全新问题,确保数据独立性并避免污染。为提升难度,设计九种预定义策略作为推理过程中的软约束,并采用强化学习联合优化结构有效性、推理复杂度与答案一致性。以自回归提示生成的推理轨迹长度反映认知复杂度,鼓励生成符合长链思维特点的难题。在五个基准测试(包括GSM8K、MATH-500、AIME2024、AIME2025、OlympiadBench)上,MathSmith在短链与长链思维设置下均持续优于现有基线。此外,弱点聚焦的变体生成模块可针对性提升特定知识点。整体显示其强可扩展性、泛化性与迁移能力,证明高难度合成数据在推动大模型推理方面的潜力。代码与数据已开源。
原文摘要 · Abstract (English)
Large language models have achieved substantial progress in mathematical reasoning, yet their advancement is limited by the scarcity of high-quality, high-difficulty training data. Existing synthesis methods largely rely on transforming human-written templates, limiting both diversity and scalability. We propose MathSmith, a novel framework for synthesizing challenging mathematical problems to enhance LLM reasoning. Rather than modifying existing problems, MathSmith constructs new ones from scratch by randomly sampling concept-explanation pairs from PlanetMath, ensuring data independence and avoiding contamination. To increase difficulty, we design nine predefined strategies as soft constraints during rationales. We further adopts reinforcement learning to jointly optimize structural validity, reasoning complexity, and answer consistency. The length of the reasoning trace generated under autoregressive prompting is used to reflect cognitive complexity, encouraging the creation of more demanding problems aligned with long-chain-of-thought reasoning. Experiments across five benchmarks, categorized as easy & medium (GSM8K, MATH-500) and hard (AIME2024, AIME2025, OlympiadBench), show that MathSmith consistently outperforms existing baselines under both short and long CoT settings. Additionally, a weakness-focused variant generation module enables targeted improvement on specific concepts. Overall, MathSmith exhibits strong scalability, generalization, and transferability, highlighting the promise of high-difficulty synthetic data in advancing LLM reasoning capabilities. Our code and data are available at https://github.com/Jasaxion/MathSmith.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。