用难解图问题训练大模型,提升复杂推理能力。
NPG-Muse: Scaling Long Chain-of-Thought Reasoning with NP-Hard Graph Problems
- 用难解图问题生成合成数据,驱动深度推理学习。
- 7B模型在图推理任务上超越32B的QwQ模型。
- 适合想提升模型逻辑与数学推理能力的研究者。
近期大语言模型在复杂推理任务上取得显著进展,主要依赖于长思维链(Long CoT)能力。然而,培养这种能力严重依赖高质量人工标注的后训练数据(如数学和代码),成本高昂且难以扩展。本文提出将NP难图问题作为新型合成训练语料,因其天然需要深度推理、广泛探索与反思策略,契合长思维链的核心特征。基于此,我们构建两阶段后训练框架:(i) 在拒绝采样后的NPH图实例上进行长思维链监督微调(SFT),显著增强推理深度;(ii) 采用细粒度奖励设计的强化学习(RL),提升推理效率。所提出的NPG-Muse系列模型在数学、编程、逻辑与图推理基准上均表现优异。NPG-Muse-7B在NPH图问题上的准确率与推理效率均超越QwQ-32B。结果表明,NPH图问题可作为高效且可扩展的资源,推动大模型长思维链推理的发展。实现代码见 https://github.com/littlewyy/NPG-Muse。
原文摘要 · Abstract (English)
Reasoning Large Language Models (RLLMs) have recently achieved remarkable progress on complex reasoning tasks, largely enabled by their long chain-of-thought (Long CoT) capabilities. However, developing these Long CoT behaviors relies heavily on post-training with high-quality datasets, which are typically costly and human-curated (e.g., mathematics and code), leaving scalable alternatives unexplored. In this work, we introduce NP-hard (NPH) graph problems as a novel synthetic training corpus, as they inherently require deep reasoning, extensive exploration, and reflective strategies, which are the core characteristics of Long CoT reasoning. Building on this insight, we develop a two-stage post-training framework: (i) Long-CoT Supervised Fine-Tuning (SFT) on rejection-sampled NPH graph instances, which substantially enhances reasoning depth, and (ii) Reinforcement Learning (RL) with a fine-grained reward design, which sharpens reasoning efficiency. The resulting NPG-Muse-series models exhibit substantially enhanced Long CoT reasoning capabilities, achieving consistent gains across mathematics, coding, logical, and graph reasoning benchmarks. NPG-Muse-7B even surpasses QwQ-32B on NPH graph problems in both accuracy and reasoning efficiency. These results position NPH graph problems as an effective and scalable resource for advancing Long CoT reasoning in LLM post-training. Our implementation is available at https://github.com/littlewyy/NPG-Muse.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。