自动构建逻辑推理任务，评估并提升大模型推理能力
SLR: Automated Synthesis for Scalable Logical Reasoning
- 自动生成推理指令、验证程序和真实规则，无需人工标注
- 创建含1.9万条提示的分级基准，揭示大模型逻辑推理缺陷
- 显著提升模型性能且计算成本远低于现有方法，适合训练优化
我们提出SLR，一个端到端框架，通过可扩展逻辑推理系统性地评估与训练大语言模型。给定用户任务描述，SLR自动合成（i）归纳推理任务的指令提示，（ii）可执行于模型输出以提供可验证奖励的验证程序，（iii）潜在的真实规则。该过程完全自动化、可扩展，无需人工标注，并能精确控制任务难度。基于此，我们构建了包含19,000个提示的SLR-Bench基准，分为20个渐进式课程层级，逐步增加关系、算术与递归复杂度。大规模评估显示，当前大模型虽常生成语法正确的规则，但逻辑推理常出错。近期推理型大模型表现虽有提升，但测试时计算开销巨大，仅1,000个提示耗时即超300美元。而通过SLR进行课程学习，可使Llama-3-8B在SLR-Bench上的准确率翻倍，达到与Gemini-Flash-Thinking相当的水平，且计算成本极低。此外，这些推理能力可泛化至多个主流基准，证明SLR在下游推理任务中的有效性。
原文摘要 · Abstract (English)
We introduce SLR, an end-to-end framework for systematic evaluation and training of Large Language Models (LLMs) via Scalable Logical Reasoning. Given a user's task specification, SLR automatically synthesizes (i) an instruction prompt for an inductive reasoning task, (ii) a validation program, executable on model outputs to provide verifiable rewards, and (iii) the latent ground-truth rule. This process is fully automated, scalable, requires no human annotations, and offers precise control over task difficulty. Using SLR, we create SLR-Bench, a benchmark comprising 19k prompts organized into 20 curriculum levels that progressively increase in relational, arithmetic, and recursive complexity. Large-scale evaluation reveals that contemporary LLMs readily produce syntactically valid rules, yet often fail at correct logical inference. Recent reasoning LLMs demonstrate improved performance but incur very high test-time computation, with costs exceeding $300 for just 1,000 prompts. Finally, curriculum learning via SLR doubles Llama-3-8B accuracy on SLR-Bench, achieving parity with Gemini-Flash-Thinking at a fraction of computational cost. Moreover, these reasoning capabilities generalize to a wide range of established benchmarks, underscoring the effectiveness of SLR for downstream reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。