用4500万论文训练的检索增强模型，帮科学家精准回答跨领域问题。
OpenScholar: Synthesizing Scientific Literature with Retrieval-augmented LMs
- 基于4500万开源论文检索，结合自反馈机制生成答案。
- 在多领域评测中正确率超GPT-4o 5%，引用准确率媲美人类专家。
- 适合科研人员快速整合文献，尤其适合需要高可信度回答的场景。
科学进步依赖于研究人员对日益增长的文献进行整合的能力。大型语言模型（LMs）能否协助这一任务？我们提出了OpenScholar，一种专门的检索增强型语言模型，通过从4500万篇开放获取论文中识别相关段落，并生成基于引用支持的回答来应对科学查询。为评估OpenScholar，我们构建了ScholarQABench——首个大规模多领域文献检索基准，包含2967个专家撰写的查询和208个长篇回答，覆盖计算机科学、物理、神经科学和生物医学。在ScholarQABench上，OpenScholar-8B的正确率比GPT-4o高5%，比PaperQA2高7%，尽管其规模更小且为开源模型。相比之下，GPT-4o在78%至90%的情况下会编造引用，而OpenScholar的引用准确率与人类专家相当。OpenScholar的数据存储、检索器和自反馈推理循环还能提升现成语言模型的表现：例如，OpenScholar-GPT4o使GPT-4o的正确率提升12%。在人类评估中，专家对OpenScholar-8B和OpenScholar-GPT4o的回答偏好率分别为51%和70%，高于专家撰写答案的偏好率（32%），显著优于GPT-4o。我们已开源全部代码、模型、数据存储、数据集及公共演示。
原文摘要 · Abstract (English)
Scientific progress depends on researchers' ability to synthesize the growing body of literature. Can large language models (LMs) assist scientists in this task? We introduce OpenScholar, a specialized retrieval-augmented LM that answers scientific queries by identifying relevant passages from 45 million open-access papers and synthesizing citation-backed responses. To evaluate OpenScholar, we develop ScholarQABench, the first large-scale multi-domain benchmark for literature search, comprising 2,967 expert-written queries and 208 long-form answers across computer science, physics, neuroscience, and biomedicine. On ScholarQABench, OpenScholar-8B outperforms GPT-4o by 5% and PaperQA2 by 7% in correctness, despite being a smaller, open model. While GPT4o hallucinates citations 78 to 90% of the time, OpenScholar achieves citation accuracy on par with human experts. OpenScholar's datastore, retriever, and self-feedback inference loop also improves off-the-shelf LMs: for instance, OpenScholar-GPT4o improves GPT-4o's correctness by 12%. In human evaluations, experts preferred OpenScholar-8B and OpenScholar-GPT4o responses over expert-written ones 51% and 70% of the time, respectively, compared to GPT4o's 32%. We open-source all of our code, models, datastore, data and a public demo.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。