arXiv:2510.16476cs.AI2025-10被引 11

用可验证的难问题训练大模型,提升其解决复杂优化题的能力。

NP-Engine: Empowering Optimization Reasoning in Large Language Models with Verifiable Synthetic NP Problems

  • 构建生成-验证-启发式三件套,实现对NP难题的可控训练与评估
  • 在新基准NP-BENCH上,小模型性能超越GPT-4o,达当前最优
  • 多任务训练让模型不仅会解难题,还能泛化到逻辑、数学等各类任务

大型语言模型在数学、编程、逻辑和谜题等任务中表现出强大的推理能力,尤其通过可验证奖励强化学习(RLVR)取得显著进展。然而,其在更复杂的优化问题,特别是NP难问题上的表现仍不充分。为此,本文提出首个针对NP难问题的综合性训练与评估框架NP-ENGINE,涵盖五个领域的10类任务,每类任务均配备可控制的实例生成器、规则验证器及启发式求解器,提供近似最优解作为真实标签。该生成-验证-启发式流水线支持分层难度下的可扩展、可验证的RLVR训练。我们还构建了基于NP-ENGINE-DATA的专用基准NP-BENCH,聚焦可行性与解质量双重评估。此外,我们推出了基于Qwen2.5-7B-Instruct的零样本RLVR模型QWEN2.5-7B-NP,通过课程学习训练,在相同模型规模下显著优于GPT-4o,达到当前最佳水平。实验还表明,基于NP-ENGINE-DATA的RLVR训练能有效提升模型在逻辑、谜题、数学和知识等非领域任务上的零样本泛化能力,且任务多样性越高,泛化能力越强。结果揭示,丰富任务驱动的RLVR训练是推动大模型推理能力发展的关键路径,并为强化学习的缩放规律提供了新见解。

原文摘要 · Abstract (English)

Large Language Models (LLMs) have shown strong reasoning capabilities, with models like OpenAI's O-series and DeepSeek R1 excelling at tasks such as mathematics, coding, logic, and puzzles through Reinforcement Learning with Verifiable Rewards (RLVR). However, their ability to solve more complex optimization problems - particularly NP-hard tasks - remains underexplored. To bridge this gap, we propose NP-ENGINE, the first comprehensive framework for training and evaluating LLMs on NP-hard problems. NP-ENGINE covers 10 tasks across five domains, each equipped with (i) a controllable instance generator, (ii) a rule-based verifier, and (iii) a heuristic solver that provides approximate optimal solutions as ground truth. This generator-verifier-heuristic pipeline enables scalable and verifiable RLVR training under hierarchical difficulties. We also introduce NP-BENCH, a benchmark derived from NP-ENGINE-DATA, specifically designed to evaluate LLMs' ability to tackle NP-hard level reasoning problems, focusing not only on feasibility but also on solution quality. Additionally, we present QWEN2.5-7B-NP, a model trained via zero-RLVR with curriculum learning on Qwen2.5-7B-Instruct, which significantly outperforms GPT-4o on NP-BENCH and achieves SOTA performance with the same model size. Beyond in-domain tasks, we demonstrate that RLVR training on NP-ENGINE-DATA enables strong out-of-domain (OOD) generalization to reasoning tasks (logic, puzzles, math, and knowledge), as well as non-reasoning tasks such as instruction following. We also observe a scaling trend: increasing task diversity improves OOD generalization. These findings suggest that task-rich RLVR training is a promising direction for advancing LLM's reasoning ability, revealing new insights into the scaling laws of RLVR.

优化推理强化学习大模型NP难题

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。