用AI预审开源项目提交，提升评审效率
AI-assisted pre-review of open-source software submissions: an experience report from BOSC 2026
- 构建AI工具自动评估六项评审标准，生成证据供人工参考
- 通过容器化测试验证项目可运行性，保障安全性和可复现性
- 人工保留最终决策权，适合希望提效的开源会议使用
多数会议依赖同行评审，但生成式AI使提交材料准备更便捷，导致投稿量激增。我们尝试让AI辅助生物信息学开源会议（BOSC）的志愿者评审员进行预审。BOSC已有一套详细的评分标准，涵盖代码公开性、许可证有效性及可运行性（即项目下载、构建和运行的难易程度）。2026年，我们开发了bosc-pre-review工具，用于评估六项评审指标，并引入Runabilly，在一次性Docker容器中构建并测试每个项目以确保安全。AI仅收集证据供人类评审员参考，所有录用决定仍由人工做出。评审结束后调查显示，大多数回复者认为预审有用，但更倾向于自行核对AI结论，而非直接采纳。
原文摘要 · Abstract (English)
Most conferences rely on peer-review of submissions, but as generative AI makes it easier than ever to prepare submission materials, some conferences are seeing an overwhelming surge of submissions. We wanted to see if generative AI could help our conference's volunteer reviewers by pre-reviewing abstracts for certain criteria. The Bioinformatics Open Source Conference (BOSC) was well-positioned to experiment with this, as we already had a detailed rubric used by reviewers to evaluate submitted abstracts on multiple criteria, including openness (public availability of the code or other content associated with the project), valid open source license, and "runnability" (how easy it is to download, build, and run the project - an important measure of reusability). For BOSC 2026, we built bosc-pre-review, an agentic skill that assessed six review criteria, and Runabilly, which builds and tests each project in a disposable Docker container for safety. The AI only gathered evidence to present to the reviewers; humans made every decision regarding the acceptance of the abstracts. After the review period, we surveyed the reviewers to determine how useful they found the pre-review. Most of those who responded said they found it useful, but they preferred to check the AI's conclusions against their own, rather than accepting the AI results unquestioningly.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。