arXiv:2504.06910cs.CL2025-04EMNLP被引 5

用数据驱动方法识别论文评审中的关键评价维度,提升评审效率与质量。

Identifying Aspects in Peer Reviews

  • 从评审文本中自动挖掘评价维度,不依赖预设评审表。
  • 构建首个带标注维度的同行评审数据集,支持社区级分析。
  • 发现评价维度选择会影响大模型生成评审检测效果,具实用价值。

同行评审是学术出版的核心环节,但投稿量激增正给流程带来压力。尽管每篇评审针对具体论文,评审者常基于如新颖性等共同评价维度进行判断,这些维度反映了学术共同体的价值观,为标准化评审流程、提升质量控制和实现计算支持提供了可能。尽管已有研究展示评价维度分析在辅助评审中的潜力,但“维度”概念仍缺乏明确定义。现有方法多依赖评审指南提取维度,而基于数据驱动的维度识别仍不充分。为此,本文提出自下而上的方法:给出维度的操作性定义,并开发一种从评审语料库中数据驱动推导维度的框架。我们构建了一个增强维度标注的同行评审数据集,展示了其在社区级评审分析中的应用。进一步表明,维度选择会显著影响下游任务,如大型语言模型生成评审的检测效果。研究成果为评价维度的严谨、数据驱动研究奠定了基础,并为自然语言处理在评审支持中的新应用铺平道路。

原文摘要 · Abstract (English)

Peer review is central to academic publishing, but the growing volume of submissions is straining the process. This motivates the development of computational approaches to support peer review. While each review is tailored to a specific paper, reviewers often make assessments according to certain aspects such as Novelty, which reflect the values of the research community. This alignment creates opportunities for standardizing the reviewing process, improving quality control, and enabling computational support. While prior work has demonstrated the potential of aspect analysis for peer review assistance, the notion of aspect remains poorly formalized. Existing approaches often derive aspects from review forms and guidelines, yet data-driven methods for aspect identification are underexplored. To address this gap, our work takes a bottom-up approach: we propose an operational definition of aspect and develop a data-driven schema for deriving aspects from a corpus of peer reviews. We introduce a dataset of peer reviews augmented with aspects and show how it can be used for community-level review analysis. We further show how the choice of aspects can impact downstream applications, such as LLM-generated review detection. Our results lay a foundation for a principled and data-driven investigation of review aspects, and pave the path for new applications of NLP to support peer review.

同行评审自然语言处理评价维度

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。