用大模型指导小模型，低成本精准分析半导体电镜图像。
Multi-Modal Instruction-Tuning Small-Scale Language-and-Vision Assistant for Semiconductor Electron Micrograph Analysis
- 用大模型生成指令数据，训练小型多模态模型完成图像理解任务。
- 在电镜图像上实现零样本问答与分类，准确率提升显著。
- 适合半导体领域专家快速构建私有化分析工具，无需大量标注数据。
我们提出一种基于视觉-语言指令调优的新框架，用于半导体制造中的电子显微镜图像分析。该框架采用独特的师生学习方法，利用GPT-4等预训练多模态大模型生成指令跟随数据，用于零样本视觉问答（VQA）和分类任务，从而定制小型多模态模型（SMMs）以适应电镜图像分析。该方法将领域专业知识从大型模型迁移至小型模型，极大减少对人工标注的依赖。本研究提供了一种安全、低成本且可定制的显微图像分析方案，解决了半导体制造中使用专有模型的挑战。
原文摘要 · Abstract (English)
We present a novel framework for analyzing and interpreting electron microscopy images in semiconductor manufacturing using vision-language instruction tuning. The framework employs a unique teacher-student approach, leveraging pre-trained multimodal large language models such as GPT-4 to generate instruction-following data for zero-shot visual question answering (VQA) and classification tasks, customizing smaller multimodal models (SMMs) for microscopy image analysis, resulting in an instruction-tuned language-and-vision assistant. Our framework merges knowledge engineering with machine learning to integrate domain-specific expertise from larger to smaller multimodal models within this specialized field, greatly reducing the need for extensive human labeling. Our study presents a secure, cost-effective, and customizable approach for analyzing microscopy images, addressing the challenges of adopting proprietary models in semiconductor manufacturing.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。