通过细粒度评估揭示大模型推理能力的优劣与训练方式的关系
Dissecting Logical Reasoning in LLMs: A Fine-Grained Evaluation and Supervision Study
- 设计三维度评估框架，分析推理过程的准确性、步骤合理性与表征
- 自然语言监督更擅长长链和分布外问题，符号化监督更利于原子步骤正确
- 发现微调主要优化生成步骤而非提前得出答案，适合研究推理机制者
逻辑推理是大语言模型的核心能力，但现有仅依赖最终答案准确率的基准无法反映推理过程质量。为此，我们提出 FineLogic，一个细粒度评估框架，从整体准确率、分步严谨性、表征层面探查三个维度评估逻辑推理。基于此框架，我们系统研究了不同微调监督形式对推理能力的影响：在四种监督风格下（一种自然语言，三种符号化）微调模型。结果表明存在关键权衡：自然语言监督在分布外和长链问题上泛化能力更强，而符号化监督更能培养结构严谨的原子推理步骤。此外，探查分析显示，微调主要优化了模型逐步生成的过程，而非提升早期收敛能力。本研究为评估和改进大模型逻辑推理提供了更严格的视角。代码已开源：https://github.com/YujunZhou/FineLogic。
原文摘要 · Abstract (English)
Logical reasoning is a core capability for large language models (LLMs), yet existing benchmarks that rely solely on final-answer accuracy fail to capture the quality of the reasoning process. To address this, we introduce FineLogic, a fine-grained evaluation framework that assesses logical reasoning across three dimensions: overall accuracy, stepwise soundness, and representation-level probing. Leveraging this framework, we conduct a comprehensive study on how different supervision formats in fine-tuning shape reasoning abilities. We fine-tune LLMs on four supervision styles: one in natural language and three symbolic variants. We find a key trade-off: natural language supervision excels at generalization to out-of-distribution and long-chain problems, whereas symbolic supervision is superior at instilling structurally sound, atomic reasoning steps. Furthermore, our probing analysis indicates that fine-tuning primarily refines the model's step-by-step generation process, rather than improving its ability to converge on an answer early. Together, our framework and analysis provide a more rigorous lens for evaluating and improving logical reasoning in LLMs. The code is available at https://github.com/YujunZhou/FineLogic.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。