arXiv:2601.04770cs.AIcs.DB2026-01被引 3

评测大模型科学推理时是否严格遵守科学规范,避免只看结果对错。

SciIF: Benchmarking Scientific Instruction Following Towards Rigorous Scientific Intelligence

  • 设计多学科基准,要求模型解题时必须满足科学条件、语义稳定性和流程规范
  • 不仅看答案对错,更检查模型是否明确提供合规证据,实现可审计评估
  • 适合希望提升模型科学可信度的研究者与开发者使用

随着大语言模型从通用知识检索转向复杂科学发现,其评估标准也需契合科学探究的严谨性。现有基准存在明显盲区:通用指令遵循评估关注表面格式,领域专用基准仅衡量最终答案正确性,常奖励结果正确但理由错误的模型。为此,我们提出科学指令遵循能力:在解题过程中严格遵守确立科学有效性的约束。具体而言,引入SciIF——一个跨学科基准,通过配对大学水平问题与固定约束清单,在三个支柱上评估该能力:科学条件(如边界检查与假设)、语义稳定性(如单位和符号规范)以及特定流程(如必需数值方法)。独特之处在于强调可审计性,要求模型显式提供约束满足证据而非隐式合规。通过同时衡量解题正确性与多约束遵守情况,SciIF可实现组合推理失败的细粒度诊断,确保大模型能在科学严格的逻辑框架中作为可靠代理运行。

原文摘要 · Abstract (English)

As large language models (LLMs) transition from general knowledge retrieval to complex scientific discovery, their evaluation standards must also incorporate the rigorous norms of scientific inquiry. Existing benchmarks exhibit a critical blind spot: general instruction-following metrics focus on superficial formatting, while domain-specific scientific benchmarks assess only final-answer correctness, often rewarding models that arrive at the right result with the wrong reasons. To address this gap, we introduce scientific instruction following: the capability to solve problems while strictly adhering to the constraints that establish scientific validity. Specifically, we introduce SciIF, a multi-discipline benchmark that evaluates this capability by pairing university-level problems with a fixed catalog of constraints across three pillars: scientific conditions (e.g., boundary checks and assumptions), semantic stability (e.g., unit and symbol conventions), and specific processes(e.g., required numerical methods). Uniquely, SciIF emphasizes auditability, requiring models to provide explicit evidence of constraint satisfaction rather than implicit compliance. By measuring both solution correctness and multi-constraint adherence, SciIF enables finegrained diagnosis of compositional reasoning failures, ensuring that LLMs can function as reliable agents within the strict logical frameworks of science.

科学推理模型评估可审计性指令遵循

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。