arXiv:2505.16368cs.LGcs.AI2025-05被引 8

用逻辑难题训练大模型推理,自动验证且难度可控。

SATURN: SAT-based Reinforcement Learning to Unleash LLMs Reasoning

  • 用布尔可满足性问题构建可扩展的强化学习任务
  • 模型在逻辑题上准确率提升14%-28%,数学编程题也显著改善
  • 适合想提升大模型推理能力的研究者和开发者

如何设计能有效激发大语言模型(LLMs)推理能力的强化学习(RL)任务仍是未解难题。现有方法(如数学、编程、构建推理任务)存在三大缺陷:(1)可扩展性差,依赖人工标注或昂贵的LLM生成数据;(2)可验证性弱,模型输出难以自动可靠验证;(3)难度控制粗略,难以实现从易到难的渐进训练。为此,我们提出Saturn——一种基于布尔可满足性(SAT)问题的强化学习框架,支持任务规模化构建、规则化验证与精确难度控制。Saturn设计了课程学习流程,通过逐步增加难度的SAT任务,从简单到复杂持续训练模型。为保证训练稳定,我们提出了机制以控制难度过渡。我们构建了Saturn-2.6k数据集,包含2,660个不同难度的SAT问题,可用于评估模型推理能力随难度的变化。将Saturn应用于DeepSeek-R1-Distill-Qwen,得到Saturn-1.5B和Saturn-7B。结果表明:(1)在SAT任务上,两者平均pass@3得分分别提升+14.0和+28.1;(2)在数学与编程任务上,分别在AIME、LiveCodeBench等基准上提升+4.9和+1.8;(3)相比当前最优方法,进一步提升+8.8%。代码、数据与模型均已开源。

原文摘要 · Abstract (English)

How to design reinforcement learning (RL) tasks that effectively unleash the reasoning capability of large language models (LLMs) remains an open question. Existing RL tasks (e.g., math, programming, and constructing reasoning tasks) suffer from three key limitations: (1) Scalability. They rely heavily on human annotation or expensive LLM synthesis to generate sufficient training data. (2) Verifiability. LLMs' outputs are hard to verify automatically and reliably. (3) Controllable Difficulty. Most tasks lack fine-grained difficulty control, making it hard to train LLMs to develop reasoning ability from easy to hard. To address these limitations, we propose Saturn, a SAT-based RL framework that uses Boolean Satisfiability (SAT) problems to train and evaluate LLMs reasoning. Saturn enables scalable task construction, rule-based verification, and precise difficulty control. Saturn designs a curriculum learning pipeline that continuously improves LLMs' reasoning capability by constructing SAT tasks of increasing difficulty and training LLMs from easy to hard. To ensure stable training, we design a principled mechanism to control difficulty transitions. We introduce Saturn-2.6k, a dataset of 2,660 SAT problems with varying difficulty. It supports the evaluation of how LLM reasoning changes with problem difficulty. We apply Saturn to DeepSeek-R1-Distill-Qwen and obtain Saturn-1.5B and Saturn-7B. We achieve several notable results: (1) On SAT problems, Saturn-1.5B and Saturn-7B achieve average pass@3 improvements of +14.0 and +28.1, respectively. (2) On math and programming tasks, Saturn-1.5B and Saturn-7B improve average scores by +4.9 and +1.8 on benchmarks (e.g., AIME, LiveCodeBench). (3) Compared to the state-of-the-art (SOTA) approach in constructing RL tasks, Saturn achieves further improvements of +8.8%. We release the source code, data, and models to support future research.

强化学习逻辑推理大模型训练课程学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。