用真人论文投稿模拟测试大模型评审员的评分动态。
Modeling LLM Agent Reviewer Dynamics in Elo-Ranked Review System
- 让不同人设的大模型评审员在会议主席协调下多轮互评。
- 引入埃洛评分后，会议主席判断更准，但评审员策略性应付。
- 适合研究智能评审系统设计或评估机制公平性的学者。
本文通过真实会议论文投稿数据，研究大语言模型（LLM）评审员在埃洛（Elo）评分系统中的互动行为。多个具有不同人格特征的LLM评审员在领域主席（Area Chair）协调下进行多轮评审交互。对比基准设置与引入埃洛评分及评审员记忆的条件，仿真结果揭示若干有趣发现：引入埃洛评分可显著提升领域主席决策准确率；同时，评审员发展出利用埃洛系统的适应性策略，却未真正提高评审质量。代码已公开于 https://github.com/hsiangwei0903/EloReview。
原文摘要 · Abstract (English)
In this work, we explore the Large Language Model (LLM) agent reviewer dynamics in an Elo-ranked review system using real-world conference paper submissions. Multiple LLM agent reviewers with different personas are engage in multi round review interactions moderated by an Area Chair. We compare a baseline setting with conditions that incorporate Elo ratings and reviewer memory. Our simulation results showcase several interesting findings, including how incorporating Elo improves Area Chair decision accuracy, as well as reviewers' adaptive review strategy that exploits our Elo system without improving review effort. Our code is available at https://github.com/hsiangwei0903/EloReview.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。