构建企业级RAG评估基准,真实还原客服问答场景
WixQA: A Multi-Dataset Benchmark for Enterprise Retrieval-Augmented Generation
- 基于Wix官网帮助中心构建三类真实用户问题数据集
- 包含200条专家撰写多步答案、6222条大模型生成对子
- 提供完整知识库快照,适合企业问答系统研发者使用
检索增强生成(RAG)是现代问答系统的核心,可基于外部知识生成有依据的回答。尽管近期进展依赖开放域数据集,但企业级问答系统需要反映日常支持中具体、领域特定的问题。评估端到端RAG系统需包含问题-答案对及对应的知识库(KB)快照。为此,我们推出WixQA基准套件,其问题数据集均基于公开发布的知识库语料构建,可全面评估检索与生成组件。WixQA包含三个不同数据集:(i) WixQA-ExpertWritten,200个真实用户查询,配有专家撰写的多步答案;(ii) WixQA-Simulated,200对由用户对话提炼的专家验证问答对;(iii) WixQA-Synthetic,6,222对由大模型生成的问答对,每对基于知识库中一篇文档系统生成。我们以MIT许可证发布知识库快照及基线结果,形成适用于真实企业环境的唯一评估基准。
原文摘要 · Abstract (English)
Retrieval-Augmented Generation (RAG) is a cornerstone of modern question answering (QA) systems, enabling grounded answers based on external knowledge. Although recent progress has been driven by open-domain datasets, enterprise QA systems need datasets that mirror the concrete, domain-specific issues users raise in day-to-day support scenarios. Critically, evaluating end-to-end RAG systems requires benchmarks comprising not only question--answer pairs but also the specific knowledge base (KB) snapshot from which answers were derived. To address this need, we introduce WixQA, a benchmark suite featuring QA datasets precisely grounded in the released KB corpus, enabling holistic evaluation of retrieval and generation components. WixQA includes three distinct QA datasets derived from Wix.com customer support interactions and grounded in a snapshot of the public Wix Help Center KB: (i) WixQA-ExpertWritten, 200 real user queries with expert-authored, multi-step answers; (ii) WixQA-Simulated, 200 expert-validated QA pairs distilled from user dialogues; and (iii) WixQA-Synthetic, 6,222 LLM-generated QA pairs, with one pair systematically derived from each article in the knowledge base. We release the KB snapshot alongside the datasets under MIT license and provide comprehensive baseline results, forming a unique benchmark for evaluating enterprise RAG systems in realistic enterprise environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。