用微调小模型自动提取工程图中的尺寸公差信息,效果优于大模型。
Fine-Tuning Vision-Language Model for Automated Engineering Drawing Information Extraction
- 微调0.23亿参数的开源视觉语言模型Florence-2
- F1分数提升52.4%,幻觉率降低43.15%
- 适合制造领域自动化图纸解析任务
几何尺寸与公差(GD&T)在制造中至关重要,用于定义零件特征的可接受偏差以确保质量与功能。然而,从二维工程图中提取GD&T信息耗时且依赖人工或半自动工具。本文提出一种基于微调Florence-2(一个开源视觉语言模型)的自动化、高效提取方法,该模型在400张带专家标注的图纸上训练。为对比,评估了两个主流闭源模型GPT-4o和Claude-3.5-Sonnet,均采用零样本设置。由于计算成本高且不切实际,闭源模型未进行微调。而Florence-2通过全参数微调,在三个不同数据增强水平下进行实验。结果表明,与表现最佳的闭源模型相比,Florence-2在精度上提升29.95%,召回率提升37.75%,F1分数提高52.40%,幻觉率降低43.15%。这证明了微调小型开源视觉语言模型在特定领域任务中的有效性,为下游制造任务提供了实用高效的自动提取方案。
原文摘要 · Abstract (English)
Geometric Dimensioning and Tolerancing (GD&T) plays a critical role in manufacturing by defining acceptable variations in part features to ensure component quality and functionality. However, extracting GD&T information from 2D engineering drawings is a time-consuming and labor-intensive task, often relying on manual efforts or semi-automated tools. To address these challenges, this study proposes an automated and computationally efficient GD&T extraction method by fine-tuning Florence-2, an open-source vision-language model (VLM). The model is trained on a dataset of 400 drawings with ground truth annotations provided by domain experts. For comparison, two state-of-the-art closed-source VLMs, GPT-4o and Claude-3.5-Sonnet, are evaluated on the same dataset. All models are assessed using precision, recall, F1-score, and hallucination metrics. Due to the computational cost and impracticality of fine-tuning large closed-source VLMs for domain-specific tasks, GPT-4o and Claude-3.5-Sonnet are evaluated in a zero-shot setting. In contrast, Florence-2, a smaller model with 0.23 billion parameters, is optimized through full-parameter fine-tuning across three distinct experiments, each utilizing datasets augmented to different levels. The results show that Florence-2 achieves a 29.95% increase in precision, a 37.75% increase in recall, a 52.40% improvement in F1-score, and a 43.15% reduction in hallucination rate compared to the best-performing closed-source model. These findings highlight the effectiveness of fine-tuning smaller, open-source VLMs like Florence-2, offering a practical and efficient solution for automated GD&T extraction to support downstream manufacturing tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。