用超图评测大模型，发现其高阶推理能力不足。
Beyond Graphs: Can Large Language Models Comprehend Hypergraphs?
- 构建首个超图基准，涵盖21500个问题和多种复杂关系。
- 引入新提示技术，使模型在结构分类上提升4%（最高9%）。
- 适合研究大模型推理、知识图谱与复杂网络的学者。
现有基准如NLGraph和GraphQA主要评估大语言模型在二元关系上的表现，忽略了真实数据中广泛存在的高阶关联。超图能有效建模超越成对关系的复杂结构，但尚未被充分探索于大模型领域。为此，我们提出LLM4Hypergraph，首个综合性基准，包含21,500个问题，覆盖八类低阶、五类高阶及两类同构任务，使用来自引文网络和蛋白质结构的真实与合成超图。我们评估了六种主流大模型，包括GPT-4o，验证了该基准在揭示模型优劣势方面的有效性。通过引入七种超图语言及两项新方法——Hyper-BAG与Hyper-COT，显著提升了高阶推理能力，在结构分类任务中平均性能提升4%（最高达9%）。本工作为将超图计算能力融入大模型奠定了基础，推动其理解复杂结构的能力。代码已开源：https://github.com/iMoonLab/LLM4Hypergraph。
原文摘要 · Abstract (English)
Existing benchmarks like NLGraph and GraphQA evaluate LLMs on graphs by focusing mainly on pairwise relationships, overlooking the high-order correlations found in real-world data. Hypergraphs, which can model complex beyond-pairwise relationships, offer a more robust framework but are still underexplored in the context of LLMs. To address this gap, we introduce LLM4Hypergraph, the first comprehensive benchmark comprising 21,500 problems across eight low-order, five high-order, and two isomorphism tasks, utilizing both synthetic and real-world hypergraphs from citation networks and protein structures. We evaluate six prominent LLMs, including GPT-4o, demonstrating our benchmark's effectiveness in identifying model strengths and weaknesses. Our specialized prompting framework incorporates seven hypergraph languages and introduces two novel techniques, Hyper-BAG and Hyper-COT, which enhance high-order reasoning and achieve an average 4% (up to 9%) performance improvement on structure classification tasks. This work establishes a foundational testbed for integrating hypergraph computational capabilities into LLMs, advancing their comprehension. The source codes are at https://github.com/iMoonLab/LLM4Hypergraph.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。