arXiv:2504.14620cs.CL2025-04被引 2

用大模型分层评估论文创新性,不训练也能精准打分。

A Hierarchical Framework for Measuring Scientific Paper Innovation via Large Language Models

  • 将论文拆成章节和问答对,逐层分析创新点。
  • 在多个数据集上超越基线方法,且可解释性强。
  • 适合科研管理、期刊评审等需要量化创新的场景。

衡量科学论文的创新性既重要又具挑战。现有基于内容的方法常忽略全文上下文,难以捕捉完整创新范围,且泛化能力不足。我们提出HSPIM,一种基于大语言模型(LLMs)的分层、无需训练的框架。通过论文→章节→问答对的分解结构,利用零样本提示实现章节分类、问题生成与加权创新评分。生成的问答对聚焦章节级创新,作为额外上下文提升评分准确性。每个文本块由LLM输出新颖性得分与置信度得分,以置信度为权重聚合得到论文级创新分。为进一步提升性能,设计双层问题结构(通用+章节特有),并采用遗传算法优化提问提示组合。此外,在细粒度创新结构下,扩展为HSPIM$^+$,可同时生成新颖性、贡献度与可行性得分及其置信度。在多个科学会议论文数据集上的综合实验表明,HSPIM在有效性、泛化性和可解释性方面均优于基线方法。演示代码已公开于https://github.com/Jasaxion/HSPIM。

原文摘要 · Abstract (English)

Measuring scientific paper innovation is both important and challenging. Existing content-based methods often overlook the full-paper context, fail to capture the full scope of innovation, and lack generalization. We propose HSPIM, a hierarchical and training-free framework based on large language models (LLMs). It introduces a Paper-to-Sections-to-QAs decomposition to assess innovation. We segment the text by section titles and use zero-shot LLM prompting to implement section classification, question-answering (QA) augmentation, and weighted innovation scoring. The generated QA pair focuses on section-level innovation and serves as additional context to improve the LLM scoring. For each chunk, the LLM outputs a novelty score and a confidence score. We use confidence scores as weights to aggregate novelty scores into a paper-level innovation score. To further improve performance, we propose a two-layer question structure consisting of common and section-specific questions, and apply a genetic algorithm to optimize the question-prompt combinations. Furthermore, under the fine-grained structure of innovation, we extend HSPIM to an HSPIM$^+$ that generates novelty, contribution, and feasibility scores with respective confidence scores. Comprehensive experiments on scientific conference paper datasets show that HSPIM outperforms baseline methods in effectiveness, generalization, and interpretability. Demo code is available at https://github.com/Jasaxion/HSPIM.

创新评估大模型零样本可解释性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。