用超算加速科学文献检索,让大模型更准地回答专业问题。
HiPerRAG: High-Performance Retrieval Augmented Generation for Scientific Insights
- 采用高吞吐文档解析与查询感知编码技术,提升多模态文献处理效率。
- 在SciQ和PubMedQA上分别达到90%和76%准确率,超越GPT-4等模型。
- 支持百万级文献检索,适合需要跨学科知识整合的研究者。
科学文献量呈指数增长,导致发现未被充分利用、研究重复及跨领域协作受限。检索增强生成(RAG)可通过提升大语言模型(LLM)在处理海量信息时的事实准确性,辅助科学家工作。但将RAG扩展至数百万篇文章时,面临文档解析与知识嵌入的高计算成本,以及对科学内容细微语义对齐的算法挑战。为此,我们提出HiPerRAG,一种基于高性能计算(HPC)的RAG流程,可索引并检索超过360万篇科学论文。核心组件包括Oreo——一种用于多模态文档解析的高吞吐模型,以及ColTrast——一种利用对比学习与晚期交互技术的查询感知编码微调算法,显著提升检索精度。HiPerRAG在现有科学问答基准及本文提出的两个新基准上表现优异,在SciQ上达90%准确率,在PubMedQA上达76%,优于领域专用模型如PubMedGPT和商业LLM如GPT-4。通过在Polaris、Sunspot和Frontier超级计算机上扩展至数千张GPU,HiPerRAG实现了百万级文档规模的RAG工作流,助力统一科学知识,推动跨学科创新。
原文摘要 · Abstract (English)
The volume of scientific literature is growing exponentially, leading to underutilized discoveries, duplicated efforts, and limited cross-disciplinary collaboration. Retrieval Augmented Generation (RAG) offers a way to assist scientists by improving the factuality of Large Language Models (LLMs) in processing this influx of information. However, scaling RAG to handle millions of articles introduces significant challenges, including the high computational costs associated with parsing documents and embedding scientific knowledge, as well as the algorithmic complexity of aligning these representations with the nuanced semantics of scientific content. To address these issues, we introduce HiPerRAG, a RAG workflow powered by high performance computing (HPC) to index and retrieve knowledge from more than 3.6 million scientific articles. At its core are Oreo, a high-throughput model for multimodal document parsing, and ColTrast, a query-aware encoder fine-tuning algorithm that enhances retrieval accuracy by using contrastive learning and late-interaction techniques. HiPerRAG delivers robust performance on existing scientific question answering benchmarks and two new benchmarks introduced in this work, achieving 90% accuracy on SciQ and 76% on PubMedQA-outperforming both domain-specific models like PubMedGPT and commercial LLMs such as GPT-4. Scaling to thousands of GPUs on the Polaris, Sunspot, and Frontier supercomputers, HiPerRAG delivers million document-scale RAG workflows for unifying scientific knowledge and fostering interdisciplinary innovation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。