arXiv:2409.06725cs.CEcs.HC2024-09被引 9

用大模型+数字孪生,让铁路缺陷检测少依赖样本,还能看没见过的毛病。

DefectTwin: When LLM Meets Digital Twin for Railway Defect Inspection

  • 用多模态大模型构建铁路缺陷分析管道,支持图文视频输入
  • 对未见缺陷实现零样本泛化,精度达0.76-0.93
  • 适配平板等消费级设备,实时反馈提升用户体验

数字孪生(DT)可对物体、流程或系统进行实时监控、仿真与预测性维护。近期大型语言模型(LLMs)的进展为传统AI系统带来变革,在工业应用如铁路缺陷检测中具有巨大潜力。传统检测需大量缺陷样本以识别模式,但样本有限易导致过拟合且对未知缺陷表现差。将预训练的LLM融入DT可缓解此问题,减少对海量数据的依赖。本文提出DefectTwin,采用多模态多模型(M^2)LLM驱动的AI流程,用于分析铁路中已知与未知的视觉缺陷。该系统使铁路检测代理能借助平板等消费电子设备完成专家级缺陷分析。多模态处理器确保输出结果易于理解,即时用户反馈机制(instaUF)提升了用户体验质量(QoE)。所提M^2 LLM在包含文本、图像和视频的多模态输入上表现出色,对已知缺陷的精度达0.76–0.93,并展现出对未知缺陷的优越零样本泛化能力。我们还在消费级设备上评估了DefectTwin的延迟、词元数量及响应实用性。据我们所知,DefectTwin是首个专为铁路缺陷检测设计的集成式LLM数字孪生系统。

原文摘要 · Abstract (English)

A Digital Twin (DT) replicates objects, processes, or systems for real-time monitoring, simulation, and predictive maintenance. Recent advancements like Large Language Models (LLMs) have revolutionized traditional AI systems and offer immense potential when combined with DT in industrial applications such as railway defect inspection. Traditionally, this inspection requires extensive defect samples to identify patterns, but limited samples can lead to overfitting and poor performance on unseen defects. Integrating pre-trained LLMs into DT addresses this challenge by reducing the need for vast sample data. We introduce DefectTwin, which employs a multimodal and multi-model (M^2) LLM-based AI pipeline to analyze both seen and unseen visual defects in railways. This application enables a railway agent to perform expert-level defect analysis using consumer electronics (e.g., tablets). A multimodal processor ensures responses are in a consumable format, while an instant user feedback mechanism (instaUF) enhances Quality-of-Experience (QoE). The proposed M^2 LLM outperforms existing models, achieving high precision (0.76-0.93) across multimodal inputs including text, images, and videos of pre-trained defects, and demonstrates superior zero-shot generalizability for unseen defects. We also evaluate the latency, token count, and usefulness of responses generated by DefectTwin on consumer devices. To our knowledge, DefectTwin is the first LLM-integrated DT designed for railway defect inspection.

数字孪生缺陷检测大模型铁路安全

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。