用检索和微调大模型诊断工业设备故障,效果优于传统方法。
Retrieval-based and Fine-tuned LLM Approaches for Industrial Asset Health Monitoring and Decision Support
- 用语义搜索和混合检索提升故障诊断匹配度
- 微调后的模型准确率显著高于零样本与少样本提示
- 适合工业AI系统开发与故障推理研究者参考
工业设施运行着大量关键设备,如泵、涡轮机和压缩机。尽管工程师可凭经验识别和诊断设备问题,但将这种推理能力迁移至计算机系统仍具挑战。本文研究了仅用检索的方法与开源大语言模型(LLM)在FailureSensorIQ基准上的故障传感器诊断推理表现,该基准由IBM Research提出,是一个多选题问答任务。在仅检索方法中,每个选项被转化为查询,基于训练数据中正确与错误记录的相似性进行评分,测试并比较了TF-IDF、BM25、语义搜索与混合搜索。在基于LLM的方法中,评估了Qwen2.5-7B-Instruct模型在零样本提示、少样本提示和QLoRA微调下的表现。结果表明,语义搜索与混合搜索优于纯关键词匹配技术,说明意义相似性对工业故障诊断更为关键。在所有基于LLM的方法中,微调模型表现最佳,显著优于零样本与少样本提示。错误分析显示,随着选项数量增加,性能下降。鲁棒性分析还发现,所有方法均对选项顺序调整、标签变更、改写及额外干扰项敏感。
原文摘要 · Abstract (English)
Industrial plants run many important machines such as pumps, turbines, and compressors. Although engineers can use their experience to identify and diagnose machine problems, transferring this reasoning ability to computer systems remains difficult. This work studies how well a retrieval-only method and an open-source large language model (LLM) perform failure-sensor diagnostic reasoning using the FailureSensorIQ benchmark, a multiple-choice question-answering task introduced by IBM Research. In the retrieval-only approach, each answer option is converted into an option-level query and scored using similar correct and incorrect records from the training data. TF-IDF, BM25, semantic search, and hybrid search are tested and compared. In the LLM-based approach, the Qwen2.5-7B-Instruct model is evaluated using zero-shot prompting, few-shot prompting, and QLoRA fine-tuning. The results show that semantic search and hybrid search perform better than pure keyword-matching techniques, indicating that meaning-based similarity is more important for industrial failure-sensor reasoning. Among the LLM-based methods, the fine-tuned model achieves the best performance and substantially improves over zero-shot and few-shot prompting. Error analysis shows that performance decreases as the number of answer options increases. Robustness analysis also shows that all methods are sensitive to option shuffling, changed labels, paraphrasing, and additional distractors.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。