用多视角推理评估科研创意,更像人类专家。
InnoEval: On Research Idea Evaluation as a Knowledge-Grounded, Multi-Perspective Reasoning Problem
- 引入异构知识搜索,动态获取多元证据支持判断
- 组建跨学科评审团,实现多维度独立评价
- 在真实论文数据集上表现优于现有方法,接近专家共识
大型语言模型的快速发展推动了科学创意的激增,但创意评估能力并未同步提升。科学评估需具备知识基础、集体讨论与多准则决策。现有方法普遍存在知识面窄、评价维度单一及大模型作为裁判的固有偏差。为此,我们将创意评估视为基于知识的多视角推理问题,提出InnoEval框架,模拟人类水平的创新评估。该框架采用异构深度知识搜索引擎,从多样在线来源动态检索并锚定证据;通过包含不同学术背景评审员的创新评审委员会,实现多维度解耦评价。我们基于权威同行评审投稿构建了综合性数据集以基准测试InnoEval。实验表明,InnoEval在点对点、成对和群体评估任务中均持续优于基线方法,其判断模式与共识高度契合人类专家。
原文摘要 · Abstract (English)
The rapid evolution of Large Language Models has catalyzed a surge in scientific idea production, yet this leap has not been accompanied by a matching advance in idea evaluation. The fundamental nature of scientific evaluation needs knowledgeable grounding, collective deliberation, and multi-criteria decision-making. However, existing idea evaluation methods often suffer from narrow knowledge horizons, flattened evaluation dimensions, and the inherent bias in LLM-as-a-Judge. To address these, we regard idea evaluation as a knowledge-grounded, multi-perspective reasoning problem and introduce InnoEval, a deep innovation evaluation framework designed to emulate human-level idea assessment. We apply a heterogeneous deep knowledge search engine that retrieves and grounds dynamic evidence from diverse online sources. We further achieve review consensus with an innovation review board containing reviewers with distinct academic backgrounds, enabling a multi-dimensional decoupled evaluation across multiple metrics. We construct comprehensive datasets derived from authoritative peer-reviewed submissions to benchmark InnoEval. Experiments demonstrate that InnoEval can consistently outperform baselines in point-wise, pair-wise, and group-wise evaluation tasks, exhibiting judgment patterns and consensus highly aligned with human experts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。