用文献数据构建评审人匹配基准，提升学术审稿质量。
exHarmony: Authorship and Citations for Benchmarking the Reviewer Assignment Problem
- 将审稿人分配问题转为检索任务，融合作者、专家与引用关系信号。
- 基于OpenAlex数据构建无标签基准，上下文嵌入模型表现最优。
- 适合研究学术推荐、智能审稿系统的学者参考。
同行评审是保障学术成果质量与可靠性的重要环节，但合适的审稿人分配仍是一大挑战。传统人工方式耗时且常导致非建设性或有偏见的评审。本文提出exHarmony（eHarmony的学术版）基准，将审稿人分配问题（RAP）重新定义为检索任务。利用OpenAlex的海量数据，融合作者特征、最相似专家及引用关系等多源信号，构建无需显式标注的标准化评估数据集。我们对多种方法进行了对比测试，包括传统的词法匹配、静态神经嵌入和上下文化神经嵌入，并引入同时评估相关性与多样性的评价指标。结果表明，虽传统方法表现尚可，但基于学术文献训练的上下文嵌入模型效果最佳。研究强调了提升审稿人分配多样性与有效性的必要性。
原文摘要 · Abstract (English)
The peer review process is crucial for ensuring the quality and reliability of scholarly work, yet assigning suitable reviewers remains a significant challenge. Traditional manual methods are labor-intensive and often ineffective, leading to nonconstructive or biased reviews. This paper introduces the exHarmony (eHarmony but for connecting experts to manuscripts) benchmark, designed to address these challenges by re-imagining the Reviewer Assignment Problem (RAP) as a retrieval task. Utilizing the extensive data from OpenAlex, we propose a novel approach that considers a host of signals from the authors, most similar experts, and the citation relations as potential indicators for a suitable reviewer for a manuscript. This approach allows us to develop a standard benchmark dataset for evaluating the reviewer assignment problem without needing explicit labels. We benchmark various methods, including traditional lexical matching, static neural embeddings, and contextualized neural embeddings, and introduce evaluation metrics that assess both relevance and diversity in the context of RAP. Our results indicate that while traditional methods perform reasonably well, contextualized embeddings trained on scholarly literature show the best performance. The findings underscore the importance of further research to enhance the diversity and effectiveness of reviewer assignments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。