用论文库+大模型自动生成领域本体的测试问题。
A RAG Approach for Generating Competency Questions in Ontology Engineering
- 基于检索增强生成，从科学论文中提取知识生成问题。
- 相比零样本提示，引入领域知识使生成问题准确率提升。
- 适合本体工程、知识图谱构建的初学者和研究者使用。
能力问题（CQ）的制定是本体开发与评估方法的核心。传统上，这一任务高度依赖领域专家和知识工程师，耗时且费力。随着大语言模型（LLMs）的出现，自动化与优化该过程成为可能。不同于以往将现有本体或知识图谱作为输入的方法，本文提出一种检索增强生成（RAG）方法，仅需一组科学论文作为领域知识库，即可利用大模型自动生成能力问题。我们评估了不同论文数量与模型温度设置对RAG性能的影响，在两个本体工程任务中使用GPT-4进行实验，并与领域专家构建的真实答案进行对比。通过精确度与一致性等指标评估，结果表明：相较于零样本提示，引入相关领域知识显著提升了大模型在具体本体任务中生成能力问题的表现。
原文摘要 · Abstract (English)
Competency question (CQ) formulation is central to several ontology development and evaluation methodologies. Traditionally, the task of crafting these competency questions heavily relies on the effort of domain experts and knowledge engineers which is often time-consuming and labor-intensive. With the emergence of Large Language Models (LLMs), there arises the possibility to automate and enhance this process. Unlike other similar works which use existing ontologies or knowledge graphs as input to LLMs, we present a retrieval-augmented generation (RAG) approach that uses LLMs for the automatic generation of CQs given a set of scientific papers considered to be a domain knowledge base. We investigate its performance and specifically, we study the impact of different number of papers to the RAG and different temperature setting of the LLM. We conduct experiments using GPT-4 on two domain ontology engineering tasks and compare results against ground-truth CQs constructed by domain experts. Empirical assessments on the results, utilizing evaluation metrics (precision and consistency), reveal that compared to zero-shot prompting, adding relevant domain knowledge to the RAG improves the performance of LLMs on generating CQs for concrete ontology engineering tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。