arXiv:2506.17374cs.CVcs.AI2025-06被引 15

将工程图转化为可执行制造知识,提升自动化水平

From Drawings to Decisions: A Hybrid Vision-Language Framework for Parsing 2D Engineering Drawings into Structured Manufacturing Knowledge

  • 融合旋转检测与视觉语言模型,精准定位复杂图纸元素
  • 在1367张图上实现93.5%的F1分数,误判率仅11.5%
  • 适合制造业数字化转型中需解析图纸的工程师和系统

从2D工程图高效准确提取关键信息是推进数字制造流程的关键。这些信息包括几何尺寸与公差(GD&T)、尺寸、材料规格及文本注释。人工提取耗时且易错,通用OCR模型因布局复杂、符号多样、文字旋转等问题常失效,导致输出不完整且不可靠。为此,我们提出一种混合视觉-语言框架,结合旋转感知目标检测模型(YOLOv11-obb)与基于Transformer的视觉语言解析器。该结构化流程先用YOLOv11-OBB定位标注并提取有向边界框(OBB)图像块,再通过微调后的轻量级视觉语言模型(VLM)解析为结构化输出。我们构建了一个包含1,367张二维机械图的数据集,涵盖九类关键信息。在该数据集上训练后,采用两个开源VLM(Donut与Florence-2)进行对比实验。经微调后,Donut表现更优,达到88.5%精度、99.2%召回率、93.5% F1分数,幻觉率11.5%。案例研究显示,所提取结构化信息可支持工艺与刀具选择等下游制造任务,验证了该框架在现代图纸解析中的实用价值。

原文摘要 · Abstract (English)

Efficient and accurate extraction of key information from 2D engineering drawings is essential for advancing digital manufacturing workflows. Such information includes geometric dimensioning and tolerancing (GD&T), measures, material specifications, and textual annotations. Manual extraction is slow and labor-intensive, while generic OCR models often fail due to complex layouts, engineering symbols, and rotated text, leading to incomplete and unreliable outputs. These limitations result in incomplete and unreliable outputs. To address these challenges, we propose a hybrid vision-language framework that integrates a rotation-aware object detection model (YOLOv11-obb) with a transformer-based vision-language parser. Our structured pipeline applies YOLOv11-OBB to localize annotations and extract oriented bounding box (OBB) patches, which are then parsed into structured outputs using a fine-tuned, lightweight vision-language model (VLM). We curate a dataset of 1,367 2D mechanical drawings annotated across nine key categories. YOLOv11-OBB is trained on this dataset to detect OBBs and extract annotation patches. These are parsed using two open-source VLMs: Donut and Florence-2. Both models are lightweight and well-suited for specialized industrial tasks under limited computational overhead. Following fine-tuning of both models on the curated dataset of image patches paired with structured annotation labels, a comparative experiment is conducted to evaluate parsing performance across four key metrics. Donut outperforms Florence-2, achieving 88.5% precision, 99.2% recall, and a 93.5% F1-score, with a hallucination rate of 11.5%. Finally, a case study demonstrates how the extracted structured information supports downstream manufacturing tasks such as process and tool selection, showcasing the practical utility of the proposed framework in modernizing 2D drawing interpretation.

工程图解析视觉语言模型制造知识工业AI

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。