arXiv:2411.11437cs.DLcs.CL2024-11

研究评审员多样性如何影响评审覆盖与冗余,发现主题和资历差异更有效。

Causal Effect of Group Diversity on Redundancy and Coverage in Peer-Reviewing

  • 从评审组多样性角度评估评审覆盖与冗余,提出新衡量标准。
  • 主题、资历、出版网络差异提升覆盖度,降低冗余;地理差异无效。
  • 出版网络多样性可单独提升视角多样性,适合优化评审分配策略。

众多科学期刊和会议对同一稿件邀请多位评审人,以获取更广泛观点并减少个体偏见。本文探讨评审员群体多样性在丰富观点、提升评审效用中的作用。提出两个评估指标:评审覆盖度(覆盖论文主要内容)与评审冗余度(避免重复信息)。假设来自多样评审员的评审具有高覆盖与低冗余。基于某会议约5000篇投稿的观察数据,开展因果分析。结果表明:主题多样、资历不同或出版网络各异的评审组能显著提升覆盖度与降低冗余;但来自不同机构或地理位置的评审组未带来覆盖度提升。除地理多样性外,其他维度(机构、资历、主题、出版网络)均减少冗余。尤其,仅出版网络多样性即可在特定评审标准内引入不同视角,降低冗余。研究从群体决策视角出发,为评审分配提供可操作的多样性维度指导。

原文摘要 · Abstract (English)

A large host of scientific journals and conferences solicit peer reviews from multiple reviewers for the same submission, aiming to gather a broader range of perspectives and mitigate individual biases. In this work, we reflect on the role of diversity in the slate of reviewers assigned to evaluate a submitted paper as a factor in diversifying perspectives and improving the utility of the peer-review process. We propose two measures for assessing review utility: review coverage -- reviews should cover most contents of the paper -- and review redundancy -- reviews should add information not already present in other reviews. We hypothesize that reviews from diverse reviewers will exhibit high coverage and low redundancy. We conduct a causal study of different measures of reviewer diversity on review coverage and redundancy using observational data from a peer-reviewed conference with approximately 5,000 submitted papers. Our study reveals disparate effects of different diversity measures on review coverage and redundancy. Our study finds that assigning a group of reviewers that are topically diverse, have different seniority levels, or have distinct publication networks leads to broader coverage of the paper or review criteria, but we find no evidence of an increase in coverage for reviewer slates with reviewers from diverse organizations or geographical locations. Reviewers from different organizations, seniority levels, topics, or publications networks (all except geographical diversity) lead to a decrease in redundancy in reviews. Furthermore, publication network-based diversity alone also helps bring in varying perspectives (that is, low redundancy), even within specific review criteria. Our study adopts a group decision-making perspective for reviewer assignments in peer review and suggests dimensions of diversity that can help guide the reviewer assignment process.

评审系统多样性因果分析覆盖冗余

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。