arXiv:2411.19203cs.CL2024-11被引 3

评测大模型在数据到文本生成中的事实一致性,发现模型越大越准,但源文与参考文本差异会降低准确性。

An Extensive Evaluation of Factual Consistency in Large Language Models for Data-to-Text Generation

  • 对比5种大模型和5个数据集,用自动评估+人工判断综合检验事实一致性
  • 模型参数越多,事实一致性越高;小模型在简单数据上表现也强
  • 源文与参考文本语义不一致时,模型更易生成错误事实,适合关注可靠生成的研究者

大语言模型(LLMs)在数据到文本生成(DTG)任务中表现出色,但在生成事实一致文本方面仍具挑战。现有研究缺乏对LLM在DTG中事实一致性的深入评估。本文通过覆盖五个主流DTG数据集(E2E、ViGGo、WikiTableText、DART、WebNLG)和五类主流模型家族(T5、BART、OPT、BLOOM、Llama 2)的广泛评估,填补这一空白。为确保评估全面性,采用四种前沿自动指标并引入人工评估。结果显示:第一,Llama 2通常在生成事实一致文本方面表现优异,而较小模型如T5和BART在数据量大、词汇多样性低的数据集上也能实现较强一致性;第二,平均变化率(AROC)表明,模型规模(可训练参数数量)增加普遍提升其在DTG中的事实一致性;第三,源-参考文本语义偏离(source-reference divergence)通常会降低模型在DTG中的事实一致性。

原文摘要 · Abstract (English)

Large Language Models (LLMs) have shown exceptional performance across various Data-to-Text Generation (DTG) tasks. However, generating factually consistent text in DTG remains challenging for LLMs. Despite this, in-depth evaluations of LLM factual consistency for DTG remain missing in the current literature. This paper addresses this gap by providing an extensive evaluation of factual consistency in LLMs for DTG. Our evaluation covers five widely used DTG datasets (E2E, ViGGo, WikiTableText, DART, and WebNLG) and five prominent LLM families (T5, BART, OPT, BLOOM, and Llama 2). To ensure a thorough evaluation of factual consistency, we use four state-of-the-art automatic metrics and include essential human assessments. Our extensive evaluations reveals three key findings regarding factual consistency in LLMs for DTG. First, Llama 2 often excels in generating factually consistent text, although smaller models like T5 and BART can achieve strong factual consistency on larger, lexically less-diverse datasets. Second, the average rate of change (AROC) indicates that increasing model size (number of model trainable parameters) generally enhances factual consistency of LLMs in DTG. Third, we observe that source-reference divergence (i.e., when the reference text diverges semantically from the source) typically reduces the factual consistency of LLMs in DTG.

大模型事实一致性数据生成评估

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。