对比开源与闭源大模型在科研问答中的表现,发现混合检索增强后,开源模型也能媲美GPT-3.5。
Comparing the Performance of LLMs in RAG-based Question-Answering: A Case Study in Computer Science Literature
- 用RAG检索增强四款开源模型和GPT-3.5,在计算机文献问答中进行对比测试。
- Mistral-7b-instruct搭配RAG在二元问题和长答案任务中表现最佳,精度超其他开源模型。
- Orca-mini响应最快,但延迟差异不影响整体性能,适合低延迟场景使用。
检索增强生成(RAG)正成为降低生成模型幻觉、提升能力的重要技术。本研究对比了四款开源大语言模型(Mistral-7b-instruct、LLaMa2-7b-chat、Falcon-7b-instruct、Orca-mini-v3-7b)与OpenAI的GPT-3.5在计算机科学文献问答任务中的表现,均采用RAG支持。评估指标包括二元问题的准确率与精确率、人工专家排序、Google Gemini AI排序以及长答案的余弦相似度。结果显示,搭配RAG的GPT-3.5在二元与长答案任务中表现优异,验证其先进性;其中,Mistral-7b-instruct在开源模型中综合表现最优。此外,Orca-mini-v3-7b平均响应延迟最短,而LLaMa2-7b-chat延迟最高。研究证明,开源模型在良好基础设施支持下可与闭源模型比肩。
原文摘要 · Abstract (English)
Retrieval Augmented Generation (RAG) is emerging as a powerful technique to enhance the capabilities of Generative AI models by reducing hallucination. Thus, the increasing prominence of RAG alongside Large Language Models (LLMs) has sparked interest in comparing the performance of different LLMs in question-answering (QA) in diverse domains. This study compares the performance of four open-source LLMs, Mistral-7b-instruct, LLaMa2-7b-chat, Falcon-7b-instruct and Orca-mini-v3-7b, and OpenAI's trending GPT-3.5 over QA tasks within the computer science literature leveraging RAG support. Evaluation metrics employed in the study include accuracy and precision for binary questions and ranking by a human expert, ranking by Google's AI model Gemini, alongside cosine similarity for long-answer questions. GPT-3.5, when paired with RAG, effectively answers binary and long-answer questions, reaffirming its status as an advanced LLM. Regarding open-source LLMs, Mistral AI's Mistral-7b-instruct paired with RAG surpasses the rest in answering both binary and long-answer questions. However, among the open-source LLMs, Orca-mini-v3-7b reports the shortest average latency in generating responses, whereas LLaMa2-7b-chat by Meta reports the highest average latency. This research underscores the fact that open-source LLMs, too, can go hand in hand with proprietary models like GPT-3.5 with better infrastructure.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。