构建电信领域技术文档问答数据集，提升检索模型性能。
Telco-DPR: A Hybrid Dataset for Evaluating Retrieval Models of 3GPP Technical Specifications
- 融合文本与表格的混合数据集，支持多模态信息检索。
- 层级检索模型DHR在Top-10准确率上达86.2%，优于传统方法。
- 适合电信、通信工程等领域的研发人员参考使用。
本文提出一个面向电信领域的问答系统，基于第三代合作伙伴计划（3GPP）技术文档构建。同时，提出一种混合数据集Telco-DPR，包含经筛选的3GPP语料库，以文本与表格混合格式呈现，并设计了合成的问答对用于评估问答系统的检索性能。采用稀疏模型（如BM25）和稠密模型（如DPR、DHR）进行对比评估，指标为Top-K准确率与平均倒数排名（MRR）。结果表明，通过在文档与段落两级微调的层级检索模型DHR表现最优，实现86.2%的Top-10准确率。此外，结合检索增强生成（RAG）技术验证了该混合数据集的优势；所提问答系统使用GPT-4与优化后的RAG模型，在相同数据集上相较先前基准提升14%的答对准确率。
原文摘要 · Abstract (English)
This paper proposes a Question-Answering (QA) system for the telecom domain using 3rd Generation Partnership Project (3GPP) technical documents. Alongside, a hybrid dataset, Telco-DPR, which consists of a curated 3GPP corpus in a hybrid format, combining text and tables, is presented. Additionally, the dataset includes a set of synthetic question/answer pairs designed to evaluate the retrieval performance of QA systems on this type of data. The retrieval models, including the sparse model, Best Matching 25 (BM25), as well as dense models, such as Dense Passage Retriever (DPR) and Dense Hierarchical Retrieval (DHR), are evaluated and compared using top-K accuracy and Mean Reciprocal Rank (MRR). The results show that DHR, a retriever model utilising hierarchical passage selection through fine-tuning at both the document and passage levels, outperforms traditional methods in retrieving relevant technical information, achieving a Top-10 accuracy of 86.2%. Additionally, the Retriever-Augmented Generation (RAG) technique, used in the proposed QA system, is evaluated to demonstrate the benefits of using the hybrid dataset and the DHR. The proposed QA system, using the developed RAG model and the Generative Pretrained Transformer (GPT)-4, achieves a 14% improvement in answer accuracy, when compared to a previous benchmark on the same dataset.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。