剖析大模型科学推理中知识与推理的作用，发现知识获取是主要瓶颈。
Demystifying Scientific Problem-Solving in LLMs by Probing Knowledge and Reasoning
- 构建科学推理多任务基准集SciReas及子集SciReas-Pro
- 揭示模型在复杂推理中依赖外部知识且显式推理能提升知识调用
- 适合研究大模型认知机制或开发科学助手的科研人员
科学问题求解对大语言模型（LLMs）构成独特挑战，需兼具深度领域知识与复杂推理能力。尽管自动科学推理系统对辅助科学家极具潜力，但目前尚无广泛采用的综合性评估基准，也缺乏系统性地分离知识与推理作用的方法。为此，我们引入SciReas——一套涵盖多种科学推理任务的基准集合，并提出其精选子集SciReas-Pro，要求更复杂的推理过程。综合评估揭示了单一基准难以呈现的科学推理性能特征。随后，我们提出KRUX探测框架，用于研究知识与推理在科学任务中的独立作用。结合两者进行深入分析，得出关键结论：（1）从模型参数中检索与任务相关知识是当前大模型科学推理的关键瓶颈；（2）在推理增强基础上，添加上下文外部知识能持续提升推理模型表现；（3）增强显式推理过程可有效提升模型调用任务相关知识的能力。
原文摘要 · Abstract (English)
Scientific problem solving poses unique challenges for LLMs, requiring both deep domain knowledge and the ability to apply such knowledge through complex reasoning. While automated scientific reasoners hold great promise for assisting human scientists, there is currently no widely adopted holistic benchmark for evaluating scientific reasoning, and few approaches systematically disentangle the distinct roles of knowledge and reasoning in these tasks. To address these gaps, we introduce SciReas, a diverse suite of existing benchmarks for scientific reasoning tasks, and SciReas-Pro, a selective subset that requires more complex reasoning. Our holistic evaluation surfaces insights about scientific reasoning performance that remain hidden when relying on individual benchmarks alone. We then propose KRUX, a probing framework for studying the distinct roles of reasoning and knowledge in scientific tasks. Combining the two, we conduct an in-depth analysis that yields several key findings: (1) Retrieving task-relevant knowledge from model parameters is a critical bottleneck for LLMs in scientific reasoning; (2) Reasoning models consistently benefit from external knowledge added in-context on top of the reasoning enhancement; (3) Enhancing verbalized reasoning improves LLMs' ability to surface task-relevant knowledge.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。