从工业技术报告构建可训练的问答数据集，提升大模型在工程场景下的推理能力。
Industrial-Instruction: An End-to-End Framework for Building Instruction-Tuning and Benchmark Datasets from Industrial Technical Reports
- 基于906份松下文档，用布局感知提取和语义索引生成问答对。
- 微调小模型后，准确率提升至42.0%，F1达63.5%。
- 对比开源与闭源模型生成数据，揭示质量与成本权衡。
工业技术报告蕴含丰富的维护、故障排查与产品工程知识，但其结构异质（密集文本、规格、表格）导致难以用标准检索与问答系统处理，且缺乏基于此类文档的公开指令微调与基准数据集。本文提出Industrial-Instruction，贡献：(i) 两个从真实工业技术报告构建的开放问答数据集；(ii) 端到端生成管道。利用906份公开松下文档（7,525页），采用布局感知提取，构建语义检索索引，并在五类查询-文档关系下（无关检索、单/多文档支持、单/多文档答案）生成基于证据的多项选择题。过滤初始23.9k样本后，每个数据集提供约13.6k QA对，附带源文档与保留测试集。微调参数少于100亿的小型开源大模型，在松下基准上，集匹配准确率从28.5%提升至42.0%，F1从46.6%升至63.5%。我们发布两版并行数据集：一由开源的Qwen3-30B-A3B-Instruct生成，一由闭源的Claude-Opus-4.6 API生成，实现开闭源模型生成数据的直接对比。后者产生更干净的原始语料，微调增益更大，但成本高出约两个数量级。MMLU评估显示，使用Claude-Opus-4.6数据训练的模型基本保留全部通用知识，而Qwen生成数据导致轻微但可测的知识遗忘。整体为从真实文档构建可扩展工业基准与训练数据提供了可复现路径。
原文摘要 · Abstract (English)
Industrial technical reports contain high-value knowledge for maintenance, troubleshooting, and product engineering, but their heterogeneous structure (dense prose, specifications, tables) makes them difficult to index and reason over with standard retrieval and QA pipelines, and no public instruction-tuning or benchmark datasets are built from such documents. We address this gap with Industrial-Instruction, contributing (i) two open QA datasets built from real industrial technical reports and (ii) the end-to-end pipeline that produces them. Using 906 public Panasonic documents (7,525 pages), we apply layout-aware extraction, build a semantic retrieval index, and synthesize multiple-choice QA grounded in retrieved evidence under five query-document relationships (irrelevant retrieval, single-/multi-document support, single-/multi-document answer). After filtering an initial 23.9k generated samples, each dataset provides approximately 13.6k QA pairs with source documents and a held-out benchmark split. Fine-tuning small open LLMs (under 10B parameters) improves Set-Match Accuracy from 28.5% to 42.0% and F1 from 46.6% to 63.5% on the Panasonic benchmark. We release two parallel versions built by the same pipeline: one generated with the open-weight Qwen3-30B-A3B-Instruct model and one with the closed, API-based Claude-Opus-4.6 model, enabling a direct comparison of open- versus frontier-model data generation. The Claude-Opus-4.6 dataset yields a cleaner raw corpus and larger fine-tuning gains, at roughly two orders of magnitude higher cost. MMLU evaluation shows models trained on the Claude-Opus-4.6 data retain essentially all general knowledge, versus a small but measurable forgetting effect for the Qwen-generated data. Together, these datasets and pipeline offer a practical, reproducible path toward scalable industrial benchmarks and training data from real-world documentation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。