arXiv:2505.19914cs.CLcs.AI2025-05NeurIPS被引 50

用可验证谜题训练大模型,显著提升逻辑推理能力。

Enigmata: Scaling Logical Reasoning in Large Language Models with Synthetic Verifiable Puzzles

  • 构建生成-验证框架,自动产难题并评估解题正确性。
  • 训练模型在多个谜题基准上超越o3-mini-high和o1。
  • 适合研究逻辑推理、强化学习与多任务训练的学者。

大型语言模型(LLM)如OpenAI的o1和DeepSeek的R1通过可验证奖励的强化学习(RLVR)在数学和编程等高级推理任务中表现优异,但在无需领域知识的人类可解谜题上仍表现不足。本文提出Enigmata,首个专用于提升模型谜题推理能力的综合性工具集,包含7个类别共36项任务,每项任务配备可控制难度的生成器和基于规则的验证器,支持大规模多任务强化学习训练、细粒度分析及无缝集成到RLVR流程。进一步提出Enigmata-Eval严格评测基准,并开发优化的多任务RLVR策略。训练模型Qwen2.5-32B-Enigmata在Enigmata-Eval、ARC-AGI(32.8%)、ARC-AGI 2(0.6%)等基准上持续优于o3-mini-high和o1,且在跨域谜题与数学推理任务中具有良好泛化性,多任务代价小。在更大模型(Seed1.5-Thinking,20B激活参数,200B总参数)上使用Enigmata数据,进一步推动AIME(2024-2025)、BeyondAIME、GPQA(Diamond)等前沿数理任务性能达到新高,展现显著泛化优势。本工作提供统一可控的逻辑推理增强框架。资源见https://seed-enigmata.github.io。

原文摘要 · Abstract (English)

Large Language Models (LLMs), such as OpenAI's o1 and DeepSeek's R1, excel at advanced reasoning tasks like math and coding via Reinforcement Learning with Verifiable Rewards (RLVR), but still struggle with puzzles solvable by humans without domain knowledge. We introduce Enigmata, the first comprehensive suite tailored for improving LLMs with puzzle reasoning skills. It includes 36 tasks across seven categories, each with 1) a generator that produces unlimited examples with controllable difficulty and 2) a rule-based verifier for automatic evaluation. This generator-verifier design supports scalable, multi-task RL training, fine-grained analysis, and seamless RLVR integration. We further propose Enigmata-Eval, a rigorous benchmark, and develop optimized multi-task RLVR strategies. Our trained model, Qwen2.5-32B-Enigmata, consistently surpasses o3-mini-high and o1 on the puzzle reasoning benchmarks like Enigmata-Eval, ARC-AGI (32.8%), and ARC-AGI 2 (0.6%). It also generalizes well to out-of-domain puzzle benchmarks and mathematical reasoning, with little multi-tasking trade-off. When trained on larger models like Seed1.5-Thinking (20B activated parameters and 200B total parameters), puzzle data from Enigmata further boosts SoTA performance on advanced math and STEM reasoning tasks such as AIME (2024-2025), BeyondAIME and GPQA (Diamond), showing nice generalization benefits of Enigmata. This work offers a unified, controllable framework for advancing logical reasoning in LLMs. Resources of this work can be found at https://seed-enigmata.github.io.

逻辑推理强化学习谜题训练多任务学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。