用图结构提升大模型对研究想法的评估能力,更准且省资源。
GraphEval: A Lightweight Graph-Based LLM Framework for Idea Evaluation
- 把想法拆成观点节点,用大模型和相似度连成图
- 比现有方法高14%以上准确率,还能识别抄袭
- 适合学术评审、创新评估等需要客观打分的场景
大语言模型在评估学术研究想法方面展现出强大潜力,但现有基于提示或微调轻量模型的方法常因不稳定、难以理解复杂语义而表现不佳。为此,我们提出GraphEval——一种轻量级图结构驱动的大模型想法评估框架。核心思路是:利用小模型提示将复杂想法分解为可理解的观点节点,并通过大模型关系抽取或BERT相似度构建节点间连接,形成观点图。在此基础上,提出两种轻量评估方法:(1)GraphEval-LP——无需训练的标签传播算法,实现评分跨节点传播;(2)GraphEval-GNN——轻量图神经网络,在极少计算资源下预测评分。为进一步提升新颖性判断能力,还引入新颖性检测模块。在两个数据集上的实验表明,GraphEval相比基线提升至少14%的F1分数,同时具备有效识别抄袭的能力,且计算与API成本低。
原文摘要 · Abstract (English)
The powerful capabilities of Large Language Models (LLMs) have led to their growing use in evaluating human-generated content, particularly in evaluating research ideas within academic settings. Existing solutions primarily rely on prompt-based LLM methods or fine-tuned lightweight language models for idea evaluation. However, these methods are often unstable and struggle to comprehend the complex semantic information embedded in the ideas, impeding their ability to perform high-quality evaluations. To address the above challenges, we propose GraphEval, a lightweight graph-based LLM framework for idea evaluation. Our insight is that a complex idea can be broken down into comprehensible viewpoint nodes using prompts from small LLMs. These viewpoint nodes can then be linked together through edges created from LLM-based relation extraction and/or BERT similarity scores. The created viewpoint-graph can be used to conveniently propagate scores across view-nodes to improve the robustness of the idea evaluations. In particular, we propose two lightweight graph-based methods for idea evaluation: (1) GraphEval-LP: a training-free label propagation algorithm that propagates evaluation scores from known view-nodes to unknown nodes; (2) GraphEval-GNN: a Graph Neural Networks (GNN) that is trained to predict the evaluation scores given the observed graph with minimal computation resources. Moreover, to overcome LLM's limitation in objectively assessing the novelty of ideas, we further propose a novelty detection model to GraphEval-GNN to enhance its capability in judging idea novelty. Experiments on two datasets show GraphEval improves F1 scores by at least 14% with low computation and API costs. Additionally, GraphEval can effectively detect plagiarized ideas.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。