用真实汽车制造数据验证了RAG在需求工程中的工业级应用效果。
An Industrial-Scale Retrieval-Augmented Generation Framework for Requirements Engineering: Empirical Evaluation with Automotive Manufacturing Data
- 基于混合语义与关键词检索的RAG框架，提升需求提取精度。
- 相比基线模型，准确率提升24.4%~19.6%，实现98.2%提取准确率。
- 适合汽车制造等工业场景中需高可追溯性的自动化需求管理。
工业4.0时代的需求工程面临技术规范、供应商清单和合规标准等异构非结构化文档的挑战。尽管检索增强生成（RAG）在知识密集型任务中表现良好，但尚未有研究基于真实工业需求工程流程与全面的生产级指标进行评估。本文使用涵盖2015–2023年四个标准（MBN 9666-1, MBN 9666-2, BQF 9666-5, MBN 9666-9）共669条需求，以及49家供应商资质及其配套文档的真实汽车制造数据，对RAG框架进行了系统性实证评估。通过与BERT基线及无约束LLM方法对比，该框架实现98.2%的需求提取准确率，且具备完整可追溯性，优于基线24.4%和19.6%。混合语义-词法检索达到0.847的MRR。专家评估平均分4.32/5.0。评估显示人工分析时间减少83%，通过多提供商LLM编排实现47%成本节约。消融实验量化各组件贡献。纵向分析揭示需求量下降55%，信息安全关注提升1800%，识别出10家（20.4%）需重新认证的旧供应商，避免潜在230万美元合同罚金。
原文摘要 · Abstract (English)
Requirements engineering in Industry 4.0 faces critical challenges with heterogeneous, unstructured documentation spanning technical specifications, supplier lists, and compliance standards. While retrieval-augmented generation (RAG) shows promise for knowledge-intensive tasks, no prior work has evaluated RAG on authentic industrial RE workflows using comprehensive production-grade performance metrics. This paper presents a comprehensive empirical evaluation of RAG for industrial requirements engineering automation using authentic automotive manufacturing documentation comprising 669 requirements across four specification standards (MBN 9666-1, MBN 9666-2, BQF 9666-5, MBN 9666-9) spanning 2015-2023, plus 49 supplier qualifications with extensive supporting documentation. Through controlled comparisons with BERT-based and ungrounded LLM approaches, the framework achieves 98.2% extraction accuracy with complete traceability, outperforming baselines by 24.4% and 19.6%, respectively. Hybrid semantic-lexical retrieval achieves MRR of 0.847. Expert quality assessment averaged 4.32/5.0 across five dimensions. The evaluation demonstrates 83% reduction in manual analysis time and 47% cost savings through multi-provider LLM orchestration. Ablation studies quantify individual component contributions. Longitudinal analysis reveals a 55% reduction in requirement volume coupled with 1,800% increase in IT security focus, identifying 10 legacy suppliers (20.4%) requiring requalification, representing potential $2.3M in avoided contract penalties.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。