构建可追踪论文修改的元审稿数据集,提升审稿透明度
Metag: A dataset to build agentic meta-reviewing capabilities

- 收集审稿前后论文版本差异,匹配作者回应与实际修改
- 包含349条高质量修改条目,精准定位修改位置
- 适合开发智能审稿助手,减轻元审稿人负担
AI工具正广泛参与科研全流程,从实验设计到同行评审。随着会议投稿量持续增长,元审稿人需整合审稿意见、作者回复和稿件修订,工作压力增大。本文提出Metag数据集,旨在加速元审稿智能体的发展,聚焦于识别审稿-回应过程中论文的具体修改。每个实例包含审稿人关切、作者提出的解决方案及对应稿件差异(diff)。数据通过获取投稿截止前与录用后两个版本的论文,计算其差异,并由人工标注者将差异与OpenReview讨论中的行动项对齐生成。最终数据集包含349条高质量行动项与论文变更关联,可支持开发方法,帮助元审稿人快速判断作者是否回应了审稿意见及其在文中的具体位置,提升同行评审的透明度与可追溯性。数据集已公开于https://github.com/microsoft/Metag-dataset。
原文摘要 · Abstract (English)
AI tools increasingly support tasks across the scientific research cycle, from experiment design and manuscript preparation to peer review. At the same time, the continuing growth in conference submissions has increased the burden on meta-reviewers, who must synthesize reviewer feedback, author rebuttals, and manuscript revisions. To address this concern, this paper introduces Metag, a dataset to accelerate the development of meta-reviewing agents, specifically to identify changes made to scientific articles during the review-rebuttal process. Each instance contains a reviewer concern, the author's proposed resolution, and the manuscript diffs implementing the stated change. Metag is collected by obtaining manuscript versions from before the review deadline and after acceptance, computing differences between the two documents, and asking human annotators to align these differences with action items from OpenReview discussions. The resulting dataset consists of 349 high-quality action items tied to paper differences and will enable building methods to empower meta reviewers to quickly identify whether authors have addressed reviewer statements and where in the paper those changes have been made, resulting in additional transparency and traceability throughout peer review. The dataset is publicly available at https://github.com/microsoft/Metag-dataset.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。