构建工程问题评测基准,揭示大模型在真实工程场景下的能力短板。
EngiBench: A Benchmark for Evaluating Large Language Models on Engineering Problem Solving
- 分三层评估工程问题求解:知识检索、上下文推理、开放式建模。
- 模型准确率随任务复杂度上升而下降,微小扰动即导致性能显著退化。
- 适合研究具身智能、可信AI的学者,关注模型鲁棒性与领域知识融合。
大型语言模型在定义明确的数学推理任务中表现优异,但现实工程问题涉及不确定性、上下文依赖和开放式情境,远超符号计算范畴。现有基准多聚焦于形式化或抽象推理,难以捕捉此类复杂性。本文提出EngiBench,一个分层评测基准,用于评估大模型在工程问题求解中的表现。该基准涵盖三个递增难度层级(基础知识检索、上下文推理、开放式建模),覆盖多个工程子领域。为深入分析模型性能,我们对每道题目系统生成三种可控变体(扰动版、知识增强版、数学抽象版),分别评估模型的鲁棒性、领域知识掌握程度及数学推理能力。实验结果表明:模型准确率随任务复杂度上升明显下降,在轻微扰动下性能显著恶化,且在高阶工程任务上仍远低于人类水平。这些发现表明当前大模型缺乏真实工程所需的高层次推理能力,亟需发展更深层、更可靠的解决问题模型。代码与数据已开源:https://github.com/AI4Engi/EngiBench。
原文摘要 · Abstract (English)
Large language models (LLMs) have shown strong performance on mathematical reasoning under well-defined conditions. However, real-world engineering problems involve uncertainty, context, and open-ended settings that extend beyond symbolic computation. Existing benchmarks largely focus on well-defined or abstract reasoning and therefore fail to capture these complexities. We introduce EngiBench, a hierarchical benchmark designed to evaluate LLMs on solving engineering problems. It spans three levels of increasing difficulty (foundational knowledge retrieval, contextual reasoning, and open-ended modeling) and covers diverse engineering subfields. To facilitate a deeper understanding of model performance, we systematically rewrite each problem into three controlled variants (perturbed, knowledge-enhanced, and math abstraction), enabling us to separately evaluate the model's robustness, domain-specific knowledge, and mathematical reasoning abilities. Experimental results show clear performance stratification across difficulty levels: model accuracy declines with task complexity, degrades under minor perturbations, and remains substantially below human performance on high-level engineering tasks. These findings reveal that current LLMs still lack the high-level reasoning needed for real-world engineering, highlighting the need for future models with deeper and more reliable problem-solving capabilities. Our source code and data are available at https://github.com/AI4Engi/EngiBench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。