构建半导体制造领域自适应RAG评估框架,解决专家依赖与评估难问题
FAB-Bench: A Framework for Adaptive RAG Benchmarking in Semiconductor Manufacturing

- 设计六项诊断指标,覆盖事实准确、上下文利用等维度
- 发现长上下文下性能退化主因是注意力稀释,存在三类不同扩展模式
- 适配多模型多框架,支持工业级RAG系统可复现评估
检索增强生成(RAG)在知识密集型应用中至关重要,但其在垂直领域的评估仍面临挑战:领域复杂、上下文尺度多样,且高度依赖成本高、不一致、不可扩展的专家评估。我们提出FAB-Bench,一个面向半导体制造领域的自适应RAG评估端到端框架。该框架定义六项诊断指标:事实准确性、上下文利用率、完整性、检索相关性、技术深度和推理一致性。通过在4K-32K token的上下文窗口上耦合检索器诊断与生成器推理分析,量化检索精度与生成保真度随上下文范围扩展的共演化关系。基于超过1,300个生成样本,我们构建了200组高质量问答对,涵盖三种合成策略:针堆找针、文档内多主题和跨文档多跳。对四个大语言模型和四个RAG框架的系统评估揭示三类不同的上下文扩展行为:对数增长、早期饱和与冷启动动态,并识别出注意力稀释是极端长上下文下性能下降的主要机制。在三个额外生产级RAG系统上的跨框架验证证实了评估方法的可迁移性。
原文摘要 · Abstract (English)
Retrieval-Augmented Generation (RAG) has become critical for knowledge-intensive applications, yet evaluating its performance in vertical domains remains difficult due to domain complexity, diverse context scales, and heavy reliance on expert assessments that are costly, inconsistent, and non-scalable. We introduce FAB-Bench, an end-to-end framework for adaptive benchmarking of RAG systems in semiconductor manufacturing. FAB-Bench defines six diagnostic metrics measuring factual accuracy, contextual utilization, completeness, retrieval relevance, technical depth, and reasoning consistency. The framework couples retriever diagnostics with generator-level reasoning analysis across context windows of 4K-32K tokens, quantifying how retrieval precision and generative fidelity co-evolve as contextual scope expands. From over 1,300 generated candidates, we curated a high-quality benchmark of 200 query-answer pairs spanning three synthesis strategies: needle-in-haystack, intra-document multi-topic, and cross-document multi-hop. Systematic evaluation across four LLMs and four RAG frameworks reveals three distinct context-scaling behaviors: logarithmic growth, early saturation, and cold-start dynamics, and identifies attention dilution as the primary mechanism behind performance degradation at extreme context lengths. Cross-framework validation on three additional production RAG systems confirms evaluation portability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。