评测大模型生成科研新点子的能力，提出可自动评估的新指标。
Can Large Language Models Unlock Novel Scientific Research Ideas?
- 设计自动化评估指标IAScore与IDI，替代人工判断
- 实验证明部分大模型能生成新颖且可行的研究设想
- 适合对AI辅助科研感兴趣的学者和开发者参考
大型语言模型（LLMs）和公开可用的ChatGPT已深刻影响AI在日常生活中的应用。本研究探讨了LLMs从科学论文中生成未来研究想法的能力。与摘要或翻译等任务不同，想法生成缺乏明确的参考标准，导致人工评估成为默认方式。然而，该过程需深厚领域知识、对论文上下文的理解及对研究前沿的认知，因而耗时、昂贵且难以扩展，尤其在新模型快速迭代背景下。目前尚无专门针对此任务的自动化评估指标。为此，本文提出两个自动化评估指标：思想对齐度（Idea Alignment Score, IAScore）与思想独特性指数（Idea Distinctness Index）。我们还进行了人工评估，以衡量生成想法的新颖性、相关性和可行性。研究揭示了LLMs在科研创意生成中的潜力与局限。我们的工作推动了对语言模型生成未来研究思路的评估与应用。数据集与代码已公开。
原文摘要 · Abstract (English)
The widespread adoption of Large Language Models (LLMs) and publicly available ChatGPT have marked a significant turning point in the integration of Artificial Intelligence (AI) into people's everyday lives. This study examines the ability of Large Language Models (LLMs) to generate future research ideas from scientific papers. Unlike tasks such as summarization or translation, idea generation lacks a clearly defined reference set or structure, making manual evaluation the default standard. However, human evaluation in this setting is extremely challenging ie: it requires substantial domain expertise, contextual understanding of the paper, and awareness of the current research landscape. This makes it time-consuming, costly, and fundamentally non-scalable, particularly as new LLMs are being released at a rapid pace. Currently, there is no automated evaluation metric specifically designed for this task. To address this gap, we propose two automated evaluation metrics: Idea Alignment Score (IAScore) and Idea Distinctness Index. We further conducted human evaluation to assess the novelty, relevance, and feasibility of the generated future research ideas. This investigation offers insights into the evolving role of LLMs in idea generation, highlighting both its capability and limitations. Our work contributes to the ongoing efforts in evaluating and utilizing language models for generating future research ideas. We make our datasets and codes publicly available
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。