用LLM提升环境科学领域文档检索精度,尤其擅长多意图查询。
How good are LLMs at Retrieving Documents in a Specific Domain?
- 构建领域专用评估数据集,自动化标注真实查询
- LLM检索在多意图查询下准确率高于Elasticsearch
- 适合需要理解复杂查询的科研数据检索场景
传统基于索引的搜索引擎依赖关键词匹配,难以捕捉用户多意图需求,尤其在环境与地球科学领域,该问题更突出。该领域研究基础设施(RIs)产生海量数据,虽已有使用Elasticsearch的集中式搜索服务,但仍面临语义理解不足的问题。为此,本文提出一种自动化方法,构建领域专属评估数据集,用于分析检索系统性能。同时引入基于大语言模型(LLM)的增强型生成检索(RAG)技术,支持自然语言查询实现高质量检索。定量与定性分析表明,在处理具有多个意图的查询时,基于LLM的检索系统相较Elasticsearch系统显著提升了结果精确度。
原文摘要 · Abstract (English)
Classical search engines using indexing methods in data infrastructures primarily allow keyword-based queries to retrieve content. While these indexing-based methods are highly scalable and efficient, due to a lack of an appropriate evaluation dataset and a limited understanding of semantics, they often fail to capture the user's intent and generate incomplete responses during evaluation. This problem also extends to domain-specific search systems that utilize a Knowledge Base (KB) to access data from various research infrastructures. Research infrastructures (RIs) from the environmental and earth science domain, which encompass the study of ecosystems, biodiversity, oceanography, and climate change, generate, share, and reuse large volumes of data. While there are attempts to provide a centralized search service using Elasticsearch as a knowledge base, they also face similar challenges in understanding queries with multiple intents. To address these challenges, we proposed an automated method to curate a domain-specific evaluation dataset to analyze the capability of a search system. Furthermore, we incorporate the Retrieval of Augmented Generation (RAG), powered by Large Language Models (LLMs), for high-quality retrieval of environmental domain data using natural language queries. Our quantitative and qualitative analysis of the evaluation dataset shows that LLM-based systems for information retrieval return results with higher precision when understanding queries with multiple intents, compared to Elasticsearch-based systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。