评测大模型在科学任务澄清中的多轮对话能力,发现其仍难处理模糊请求。
SCICONVBENCH: Benchmarking LLMs on Multi-Turn Clarification for Task Formulation in Computational Science

- 构建跨四领域的科学任务澄清基准,聚焦信息补全与矛盾检测。
- 顶尖模型仅解决52.7%的流体力学模糊问题,矛盾检测表现较好。
- 揭示模型常无声假设并隐式修正,缺乏对话依据,适合科研助手研究者使用。
大型语言模型(LLMs)正越来越多地被用作科学AI助手,现有基准主要评估其知识检索、推理、代码生成和工具使用能力,但通常假设科学问题已明确表述。然而,在实际科学协助中,用户请求常不明确,需通过多轮对话澄清后才能可靠开展计算、分析或实验。我们提出SCICONVBENCH,一个面向流体力学、固体力学、材料科学和偏微分方程(PDEs)四个计算科学领域的多轮澄清任务基准。该基准旨在评估两种互补能力:获取缺失信息(消歧)和识别并修正包含内部矛盾的错误请求(不一致化解)。基准结合结构化任务本体与基于评分标准的评估框架,系统衡量模型在澄清行为、对话一致性及最终规范准确性三个维度的表现。当前前沿模型在不一致化解方面表现尚可,但在流体力学领域仅能解决52.7%的消歧案例。此外,我们发现前沿模型频繁做出无声假设,并进行未基于对话的隐式规范修复。SCICONVBENCH为评估可靠计算科学助手所需的上游对话推理奠定了基础。代码与数据见https://github.com/csml-rpi/SciConvBench。
原文摘要 · Abstract (English)
Large Language Models (LLMs) are increasingly deployed as scientific AI as- sistants, and a growing body of benchmarks evaluates their capabilities across knowledge retrieval, reasoning, code generation, and tool use. These evaluations, however, typically assume the scientific problem is already well-posed, whereas practical scientific assistance often begins with an ill-posed user request that must be refined through dialogue before any computation, analysis, or experiment can be carried out reliably. We introduce SCICONVBENCH, a benchmark for multi- turn clarification in scientific task formulation across four computational science problem domains: fluid mechanics, solid mechanics, materials science, and par- tial differential equations (PDEs). SCICONVBENCH targets two complementary capabilities: eliciting missing information (disambiguation) and detecting and correcting erroneous requests containing internally contradictory information (in- consistency resolution). Our benchmark pairs a structured task ontology with a rubric-based evaluation framework, enabling systematic measurement of LLM per- formance across three dimensions: clarification behavior, conversational grounding, and final-specification fidelity. Current frontier models perform relatively well on inconsistency resolution, but even the best model resolves only 52.7% of the disambiguation cases in fluid mechanics. We further find that frontier LLMs fre- quently make silent assumptions and perform implicit specification repairs that are not grounded in the conversation with users. SCICONVBENCH establishes a foundation for evaluating the upstream conversational reasoning that a reliable computational science assistant requires. The code and data can be found at https://github.com/csml-rpi/SciConvBench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。