为制造业打造高精度多模态评估数据集,发现知识不足是模型瓶颈。
FORGE: Fine-grained Multimodal Evaluation for Manufacturing Scenarios

- 构建融合2D图像与3D点云的细粒度标注数据集,包含具体型号等语义信息。
- 18个先进多模态模型在三项制造任务中表现差距显著,准确率最高仅达74.3%。
- 实验证明领域知识缺乏是主因,可用该数据集微调模型提升90.8%性能。
制造业正逐步采用多模态大语言模型(MLLMs)实现从感知到自主执行的转变,但现有评估无法反映真实工业环境的严苛要求。研究受限于数据稀缺及现有数据集缺乏细粒度领域语义。为此,我们提出FORGE:首先构建高质量多模态数据集,融合真实世界2D图像与3D点云,并标注细粒度领域语义(如具体型号)。随后对18个前沿MLLM在三项制造任务(工件验证、结构表面检测、装配验证)上进行评估,揭示显著性能差异。反常规发现,视觉定位并非主要瓶颈;真正限制因素是领域知识不足,为未来研究指明方向。此外,结构化标注可直接用于训练:在3B参数小模型上进行监督微调,在未见制造场景下准确率最高提升90.8%,初步验证了构建领域适配型制造业MLLM的可行性。代码与数据集见https://ai4manufacturing.github.io/forge-web。
原文摘要 · Abstract (English)
The manufacturing sector is increasingly adopting Multimodal Large Language Models (MLLMs) to transition from simple perception to autonomous execution, yet current evaluations fail to reflect the rigorous demands of real-world manufacturing environments. Progress is hindered by data scarcity and a lack of fine-grained domain semantics in existing datasets. To bridge this gap, we introduce FORGE. Wefirst construct a high-quality multimodal dataset that combines real-world 2D images and 3D point clouds, annotated with fine-grained domain semantics (e.g., exact model numbers). We then evaluate 18 state-of-the-art MLLMs across three manufacturing tasks, namely workpiece verification, structural surface inspection, and assembly verification, revealing significant performance gaps. Counter to conventional understanding, the bottleneck analysis shows that visual grounding is not the primary limiting factor. Instead, insufficient domain-specific knowledge is the key bottleneck, setting a clear direction for future research. Beyond evaluation, we show that our structured annotations can serve as an actionable training resource: supervised fine-tuning of a compact 3B-parameter model on our data yields up to 90.8% relative improvement in accuracy on held-out manufacturing scenarios, providing preliminary evidence for a practical pathway toward domain-adapted manufacturing MLLMs. The code and datasets are available at https://ai4manufacturing.github.io/forge-web.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。