用大模型生成高质量关系抽取数据集,提升多样性和复杂度
AmalREC: A Dataset for Relation Extraction and Classification Leveraging Amalgamation of Large Language Models
- 通过五阶段流程结合模板与大模型生成句子
- 构建评估指标SEI并排序生成结果,最终融合最优句式
- 新数据集含255种关系类型,适合训练和评测关系分类模型
现有关系分类与抽取数据集常受限于关系类型单一和领域偏差。本文提出一种通用框架,利用大语言模型(LLMs)从给定关系三元组生成结构良好句子。研究聚焦四个问题:(i) 如何从关系三元组生成句子;(ii) 如何比较与排序生成结果;(iii) 能否融合各方法优势,生成更优句子;(iv) 如何评估最终数据集。针对第一个问题,采用五阶段多维度流程,结合大模型与模板引导生成。引入句子评估指数(SEI),综合语法正确性、流畅性、人类情感对齐、准确性与复杂度等维度。为解决第二问,设计SEI-Ranker模块,基于SEI筛选优质候选句。最终通过策略性融合生成高质量句子。在测试集上包含15,000条句子,训练集约150,000条,涵盖255种关系类型,显著提升关系多样性与复杂度。该工作不仅提供一个全面的新基准数据集,还对比了不同大模型在从关系三元组生成高质量句子上的表现。
原文摘要 · Abstract (English)
Existing datasets for relation classification and extraction often exhibit limitations such as restricted relation types and domain-specific biases. This work presents a generic framework to generate well-structured sentences from given tuples with the help of Large Language Models (LLMs). This study has focused on the following major questions: (i) how to generate sentences from relation tuples, (ii) how to compare and rank them, (iii) can we combine strengths of individual methods and amalgamate them to generate an even bette quality of sentences, and (iv) how to evaluate the final dataset? For the first question, we employ a multifaceted 5-stage pipeline approach, leveraging LLMs in conjunction with template-guided generation. We introduce Sentence Evaluation Index(SEI) that prioritizes factors like grammatical correctness, fluency, human-aligned sentiment, accuracy, and complexity to answer the first part of the second question. To answer the second part of the second question, this work introduces a SEI-Ranker module that leverages SEI to select top candidate generations. The top sentences are then strategically amalgamated to produce the final, high-quality sentence. Finally, we evaluate our dataset on LLM-based and SOTA baselines for relation classification. The proposed dataset features 255 relation types, with 15K sentences in the test set and around 150k in the train set organized in, significantly enhancing relational diversity and complexity. This work not only presents a new comprehensive benchmark dataset for RE/RC task, but also compare different LLMs for generation of quality sentences from relational tuples.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。