arXiv:2409.00159cs.CLcs.AI2024-09被引 4

测试大模型生成与复述图结构的能力,发现幻觉程度可反映模型优劣。

LLMs Prompted for Graphs: Hallucinations and Generative Capabilities

  • 用图谱任务评估大模型的复述与生成能力,引入幻觉度量
  • 生成随机图表现良好且可重复,暗示潜在涌现能力
  • 为网络科学与机器学习交叉研究提供新基准

大型语言模型(LLMs)现被广泛用于各类任务。本文研究其在复述和生成图结构方面的能力。首先考察模型对文献中已知图(如卡扎特俱乐部图或图谱)的复述能力;其次通过要求生成埃拉多斯-雷尼随机图,探究模型的生成潜力。鉴于训练数据中可能包含部分随机图,该任务旨在探索模型是否具备生成新图的新兴能力。针对两项任务,我们提出一种基于幻觉(即错误信息被当作事实返回)的误差评估指标。结果表明,图幻觉的幅度可表征部分模型的优越性:在复述任务中,幻觉程度与基于10,000次提示的幻觉排行榜高度相关。在生成任务中,多数模型展现出令人惊讶的高质量且可重复的结果。我们认为这为深入研究该新兴能力奠定了基础,并提出了一个具有挑战性的改进基准。两项发现促进了网络科学与机器学习领域的融合。

原文摘要 · Abstract (English)

Large Language Models (LLMs) are nowadays prompted for a wide variety of tasks. In this article, we investigate their ability in reciting and generating graphs. We first study the ability of LLMs to regurgitate well known graphs from the literature (e.g. Karate club or the graph atlas)4. Secondly, we question the generative capabilities of LLMs by asking for Erdos-Renyi random graphs. As opposed to the possibility that they could memorize some Erdos-Renyi graphs included in their scraped training set, this second investigation aims at studying a possible emergent property of LLMs. For both tasks, we propose a metric to assess their errors with the lens of hallucination (i.e. incorrect information returned as facts). We most notably find that the amplitude of graph hallucinations can characterize the superiority of some LLMs. Indeed, for the recitation task, we observe that graph hallucinations correlate with the Hallucination Leaderboard, a hallucination rank that leverages 10, 000 times more prompts to obtain its ranking. For the generation task, we find surprisingly good and reproducible results in most of LLMs. We believe this to constitute a starting point for more in-depth studies of this emergent capability and a challenging benchmark for their improvements. Altogether, these two aspects of LLMs capabilities bridge a gap between the network science and machine learning communities.

大模型图生成幻觉检测

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。