arXiv:2604.01532cs.AI2026-04被引 3

评测大模型在工业健康预测中的可靠性,发现工具调用顺序比指令理解更关键。

PHMForge: Evaluating LLM Agents on Industrial Prognostics through MCP-Native, Algorithm-Grounded Tools

  • 基于MCP协议构建真实工业场景的评估环境,分离工具使用与检索误差。
  • 最强模型在电池类任务中达80.8%成功率,但失败主因是流程编排错误。
  • 揭示静态检索无法替代动态工具执行,小模型更易出现格式错误。

大模型代理正通过模型上下文协议(MCP)调用工业资产管理工具,但其在安全关键的故障预测与健康管理(PHM)任务中的可靠性尚未可知。现有基准混淆了协议熟练度与推理能力、设备故障与代理失误、工具使用与工具获取。我们提出PHMForge,一个可闭合上述混淆的评估环境。该环境包含99个由领域专家撰写的工业场景,覆盖旋转机械、航空发动机和锂离子电池等八类资产,基于NASA PCoE等公开数据集,通过39个封装了已发表算法(如C-MAPSS、ISO 10816、Arrhenius容量衰减模型、时间序列基础模型)的MCP原生工具实现。在30个分层抽样的旋转机械/航空发动机场景中,克里普肯多夫α系数为[0.74, 0.82];电池扩展部分为单评分者。在三种代理框架和六种LLM基座上,最优配置达到80.8% pass@1,剩余差距集中于编排与工具调用顺序错误。关键消融实验显示,将MCP执行替换为基于遥测等效证据的文本检索增强生成(RAG),导致电池类任务的Remaining Useful Life pass-all-3从100%(5/5)暴跌至20%(1/5),暴露出静态检索在预测计算中的结构性局限。轨迹分解表明,编排错误是跨模型失败的主要原因,而无效工具调用集中在较小的开源模型中。前沿大模型更擅长调用工具,而非规划何时调用。

原文摘要 · Abstract (English)

LLM agents are beginning to invoke industrial asset-management tools through the Model Context Protocol (MCP), yet whether they can act reliably on this substrate for safety-critical \emph{Prognostics and Health Management (PHM)} is unanswered. Prior benchmarks conflate protocol fluency with reasoning, instrumentation failures with agent failures, and tool use with tool retrieval. We introduce \textbf{PHMForge}, an evaluation environment that closes each conflation. PHMForge ships 99 SME-authored scenarios across eight industrial asset classes spanning rotating equipment, aero-engines, and lithium-ion cells, on public datasets including NASA PCoE, served through 39 MCP-native tools wrapping published PHM algorithms (C-MAPSS, ISO~10816, Arrhenius capacity-fade models, time-series foundation models). Krippendorff's $α\in [0.74,\,0.82]$ on a 30-scenario stratified rotating-equipment/aero-engine sample; the battery extension is single-rater. Across three agentic frameworks and six LLM backbones, the strongest configuration reaches \textbf{80.8\% pass@1}, with the residual gap concentrated in orchestration and tool-sequencing errors. Crucially, an architectural ablation shows that replacing MCP execution with text-based Retrieval-Augmented Generation (RAG) over telemetry-equivalent evidence collapses Remaining Useful Life \emph{pass-all-3} from \textbf{100\% to 20\%} (5/5 vs.\ 1/5) on the battery class, exposing the structural limits of static retrieval for prognostic computation. Trajectory decomposition shows orchestration errors dominate failures across backbones, while schema-invalid tool calls concentrate in smaller open-weight models. Frontier LLMs are stronger at calling tools than at planning when to call them. PHMForge is open-sourced with deterministic evaluators, a public leaderboard, and a datasheet.

大模型评估工业预测MCP协议工具调用

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。