测试大模型在机械设计领域的表现，发现其语义理解有潜力但仍有局限。
On the effectiveness of Large Language Models in the mechanical design domain
- 用装配体名称和零件名构建无监督任务评估模型
- 零样本分类达38.6%准确率，二分类任务达62%准确率
- 针对过拟合优化模型结构，提升领域适应性
本文旨在探究大语言模型在机械工程领域的表现。我们利用ABC数据集中的语义信息，包括设计师为整体装配体命名的装配名称及各零件的语义名称。经预处理后，构建了两个无监督任务以评估不同模型架构在领域特定数据上的表现：二元句子对分类任务和零样本分类任务。通过调整学习率、丢弃率、序列长度并添加多头注意力层来缓解过拟合，微调后的模型在二元句子对分类任务中达到0.62的准确率。在零样本分类任务中，模型显著优于基线，取得0.386的顶1分类准确率。结果揭示了该领域语言学习中的特定失效模式。
原文摘要 · Abstract (English)
In this work, we seek to understand the performance of large language models in the mechanical engineering domain. We leverage the semantic data found in the ABC dataset, specifically the assembly names that designers assigned to the overall assemblies, and the individual semantic part names that were assigned to each part. After pre-processing the data we developed two unsupervised tasks to evaluate how different model architectures perform on domain-specific data: a binary sentence-pair classification task and a zero-shot classification task. We achieved a 0.62 accuracy for the binary sentence-pair classification task with a fine-tuned model that focuses on fighting over-fitting: 1) modifying learning rates, 2) dropout values, 3) Sequence Length, and 4) adding a multi-head attention layer. Our model on the zero-shot classification task outperforms the baselines by a wide margin, and achieves a top-1 classification accuracy of 0.386. The results shed some light on the specific failure modes that arise when learning from language in this domain.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。