首个工业异常检测多模态大模型评测基准，揭示现有模型离实际需求仍有差距。
MMAD: A Comprehensive Benchmark for Multimodal Large Language Models in Industrial Anomaly Detection
- 构建涵盖7类子任务的评测框架，生成39,672个问题对应8,366张工业图像
- 主流商业模型平均准确率仅74.9%，远低于工业应用要求
- 提出两种无训练优化策略，为提升工业场景表现提供新方向
在工业检测领域，多模态大语言模型（MLLMs）凭借强大的语言理解与泛化能力，有潜力革新实际应用范式。然而，尽管其在多个领域展现出优异的问题解决能力，目前对MLLMs在工业异常检测中的表现尚缺乏系统研究。为此，我们提出MMAD，首个全面覆盖工业异常检测的MLLMs基准。定义了7项关键子任务，并设计新流程生成包含39,672个问题的MMAD数据集，对应8,366张工业图像。基于该数据集，我们对多种前沿MLLMs进行了全面定量评估。结果显示，商业模型表现最佳，GPT-4o平均准确率达74.9%，但仍远低于工业应用标准。分析表明，当前MLLMs在回答工业缺陷相关问题上仍有显著提升空间。此外，我们探索了两种无需训练的性能增强策略，验证其在工业场景中的潜在价值。
原文摘要 · Abstract (English)
In the field of industrial inspection, Multimodal Large Language Models (MLLMs) have a high potential to renew the paradigms in practical applications due to their robust language capabilities and generalization abilities. However, despite their impressive problem-solving skills in many domains, MLLMs' ability in industrial anomaly detection has not been systematically studied. To bridge this gap, we present MMAD, the first-ever full-spectrum MLLMs benchmark in industrial Anomaly Detection. We defined seven key subtasks of MLLMs in industrial inspection and designed a novel pipeline to generate the MMAD dataset with 39,672 questions for 8,366 industrial images. With MMAD, we have conducted a comprehensive, quantitative evaluation of various state-of-the-art MLLMs. The commercial models performed the best, with the average accuracy of GPT-4o models reaching 74.9%. However, this result falls far short of industrial requirements. Our analysis reveals that current MLLMs still have significant room for improvement in answering questions related to industrial anomalies and defects. We further explore two training-free performance enhancement strategies to help models improve in industrial scenarios, highlighting their promising potential for future research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。