自动优化提示词，让大模型更准地从文本中提取知识图谱三元组。
Automatic Prompt Optimization for Knowledge Graph Construction: Insights from an Empirical Study
- 用自动提示优化技术生成适合三元组抽取的提示词。
- 在复杂度高、文本长时效果提升明显，优于人工设计提示。
- 适合想提升知识图谱构建效率的研究者和工程师。
知识图谱（KG）是实体及其关系的网络，广泛应用于语义搜索、推理、决策、自然语言处理、机器学习和推荐系统。从文本中抽取三元组（主语-关系-宾语）是构建知识图谱的基础，已有研究涵盖从早期的ACE 2002到近期的WebNLG 2020、REBEL和SynthIE等基准数据集。尽管大模型（LLM）被用于知识图谱构建，但手工设计任务特定提示词耗时且易受模型变化影响。近期自然语言处理研究（如自主生成）采用自动提示优化技术，基于输入输出样例生成最优或近优提示。本实证研究通过实验基准测试，探索自动提示优化在三元组抽取任务中的应用。评估了多种设置：（a）提示策略，（b）用于提示优化与任务执行的LLM，（c）模式中的标准关系数量（模式复杂度），（d）输入文本长度与多样性，（e）驱动提示优化的指标，以及（f）训练与测试所用数据集。对比三种自动提示优化器：DSPy、APE和TextGrad，使用两个三元组抽取数据集：SynthIE和REBEL。通过严格实证评估，主要贡献表明自动提示优化技术可生成接近人工水平的合理提示，从而在三元组抽取中取得更好效果，尤其在模式复杂度和文本规模增加时表现更优。
原文摘要 · Abstract (English)
A KG represents a network of entities and illustrates relationships between them. KGs are used for various applications, including semantic search and discovery, reasoning, decision-making, natural language processing, machine learning, and recommendation systems. Triple (subject-relation-object) extraction from text is the fundamental building block of KG construction and has been widely studied, for example, in early benchmarks such as ACE 2002 to more recent ones, such as WebNLG 2020, REBEL and SynthIE. While the use of LLMs is explored for KG construction, handcrafting reasonable task-specific prompts for LLMs is a labour-intensive exercise and can be brittle due to subtle changes in the LLM models employed. Recent work in NLP tasks (e.g. autonomy generation) uses automatic prompt optimization/engineering to address this challenge by generating optimal or near-optimal task-specific prompts given input-output examples. This empirical study explores the application of automatic prompt optimization for the triple extraction task using experimental benchmarking. We evaluate different settings by changing (a) the prompting strategy, (b) the LLM being used for prompt optimization and task execution, (c) the number of canonical relations in the schema (schema complexity), (d) the length and diversity of input text, (e) the metric used to drive the prompt optimization, and (f) the dataset being used for training and testing. We evaluate three different automatic prompt optimizers, namely, DSPy, APE, and TextGrad and use two different triple extraction datasets, SynthIE and REBEL. Through rigorous empirical evaluation, our main contribution highlights that automatic prompt optimization techniques can generate reasonable prompts similar to humans for triple extraction. In turn, these optimized prompts achieve improved results, particularly with increasing schema complexity and text size.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。