用视觉语言模型自动解析复杂工程图,提升制造信息提取效率。
A Multi-Stage Hybrid Framework for Automated Interpretation of Multi-View Engineering Drawings Using Vision Language Model
- 分三阶段:布局分割、符号定位、语义解析,结合检测与视觉语言模型
- 数值解析准确率达96.3%,文字识别F1为0.672,支持多视图图纸分析
- 输出统一JSON格式,适合集成到CAD和制造数据库系统
工程图是制造沟通的核心,承载设计意图、公差及生产细节。但因版式多样、方向不一、图文混合,传统人工方法、通用OCR或深度学习难以有效解析复杂多视图图纸。本文提出一种三阶段混合框架,利用现代检测模型与视觉语言模型(VLM)实现2D多视图工程图的自动化解读。第一阶段采用YOLOv11-det进行布局分割,定位视图、标题栏和注释区域;第二阶段使用YOLOv11-obb实现定向精细检测,识别尺寸、形位公差(GD&T)符号及表面粗糙度标记;第三阶段采用两个无OCR的Donut-based VLM:Alphabetical VLM从标题栏和注释中提取文本与类别信息,Numerical VLM解析尺寸、GD&T框格及粗糙度等定量数据。构建了两个专用数据集:1,000张用于布局检测,1,406张用于标注级训练。Alphabetical VLM总体F1为0.672,Numerical VLM达0.963,表明在文本与量化信息解析上表现优异。最终输出统一的JSON格式,便于与CAD和制造数据库无缝对接,提供可扩展的智能工程图分析方案。
原文摘要 · Abstract (English)
Engineering drawings are fundamental to manufacturing communication, serving as the primary medium for conveying design intent, tolerances, and production details. However, interpreting complex multi-view drawings with dense annotations remains challenging using manual methods, generic optical character recognition (OCR) systems, or traditional deep learning approaches, due to varied layouts, orientations, and mixed symbolic-textual content. To address these challenges, this paper proposes a three-stage hybrid framework for the automated interpretation of 2D multi-view engineering drawings using modern detection and vision language models (VLMs). In the first stage, YOLOv11-det performs layout segmentation to localize key regions such as views, title blocks, and notes. The second stage uses YOLOv11-obb for orientation-aware, fine-grained detection of annotations, including measures, GD&T symbols, and surface roughness indicators. The third stage employs two Donut-based, OCR-free VLMs for semantic content parsing: the Alphabetical VLM extracts textual and categorical information from title blocks and notes, while the Numerical VLM interprets quantitative data such as measures, GD&T frames, and surface roughness. Two specialized datasets were developed to ensure robustness and generalization: 1,000 drawings for layout detection and 1,406 for annotation-level training. The Alphabetical VLM achieved an overall F1 score of 0.672, while the Numerical VLM reached 0.963, demonstrating strong performance in textual and quantitative interpretation, respectively. The unified JSON output enables seamless integration with CAD and manufacturing databases, providing a scalable solution for intelligent engineering drawing analysis.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。