用逻辑验证强化大模型推理，让答案不仅对还理得通。
LogicReward: Incentivizing LLM Reasoning via Step-Wise Logical Supervision
- 通过定理证明器逐步检验推理逻辑正确性
- 8B模型在推理任务上超越GPT-4o 11.6%、o4-mini 2%
- 无需真值标签也能提供可靠训练信号
尽管大语言模型具备强大的推理能力，现有训练方法主要依赖结果反馈，可能导致答案正确但推理过程有误。已有工作对中间步骤进行监督，但仍无法保证逻辑严密性，而这在高风险场景中至关重要。为此，我们提出LogicReward，一种通过定理证明器强制执行步骤级逻辑正确的新型奖励机制。同时引入带软合一的自动形式化技术，降低自然语言歧义，提升形式化质量，使定理证明器更有效。使用LogicReward构建数据训练的8B模型，在自然语言推理和逻辑推理任务上分别优于GPT-4o 11.6%和o4-mini 2%，且训练流程简单。进一步分析表明，LogicReward提升了推理忠实度，增强了对数学和常识推理等未见任务的泛化能力，并在无真值标签时仍能提供可靠奖励信号。相关数据与代码将开源。
原文摘要 · Abstract (English)
Although LLMs exhibit strong reasoning capabilities, existing training methods largely depend on outcome-based feedback, which can produce correct answers with flawed reasoning. Prior work introduces supervision on intermediate steps but still lacks guarantees of logical soundness, which is crucial in high-stakes scenarios where logical consistency is paramount. To address this, we propose LogicReward, a novel reward system that guides model training by enforcing step-level logical correctness with a theorem prover. We further introduce Autoformalization with Soft Unification, which reduces natural language ambiguity and improves formalization quality, enabling more effective use of the theorem prover. An 8B model trained on data constructed with LogicReward surpasses GPT-4o and o4-mini by 11.6\% and 2\% on natural language inference and logical reasoning tasks with simple training procedures. Further analysis shows that LogicReward enhances reasoning faithfulness, improves generalizability to unseen tasks such as math and commonsense reasoning, and provides a reliable reward signal even without ground-truth labels. We will release all data and code at https://llm-symbol.github.io/LogicReward.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。