构建首个计算机科学系统综述数据集，支持自动化检索与筛选研究。
A Collection of Systematic Reviews in Computer Science
- 收集1212篇计算机科学系统综述，含原始布尔查询与文献信息。
- 对比零样本大模型生成查询与传统检索方法，发现精度差异显著。
- 适合做检索自动化、综述生成和可复现实验的研究者使用。
系统综述是合成科学证据的标准方法，但其构建需大量人工工作，尤其在文献检索与筛选阶段。尽管已有研究尝试自动化这些步骤，但评估资源仍主要集中于生物医学领域，限制了其他领域的可复现研究。本文提出SR4CS，一个大规模计算机科学系统综述集合，旨在支持布尔查询生成、检索与筛选的可复现研究。该语料库包含1,212篇系统综述，附带原始专家设计的布尔搜索查询、104,316条已标记参考文献及结构化方法学元数据。为便于控制评估，原始布尔查询还以归一化近似形式提供，仅作用于标题和摘要。通过基线实验，比较了近似专家查询与零样本大模型生成查询、BM25及密集检索在统一评估设置下的表现。结果揭示不同检索范式在精确率、召回率与排序行为上的系统性差异，并暴露了简单零样本布尔查询生成的局限性。SR4CS已在Zenodo（https://doi.org/10.5281/zenodo.17163932）以开放许可发布，配套文档与代码见GitHub（https://github.com/webis-de/scolia26-sr4cs），支持未来可复现的自动化综述研究。
原文摘要 · Abstract (English)
Systematic reviews are the standard method for synthesizing scientific evidence, but their creation requires substantial manual effort, particularly during retrieval and screening. While recent work has explored automating these steps, evaluation resources remain largely confined to the biomedical domain, limiting reproducible experimentation in other domains. This paper introduces SR4CS, a large-scale collection of systematic reviews in computer science, designed to support reproducible research on Boolean query generation, retrieval, and screening. The corpus comprises 1,212 systematic reviews with their original expert-designed Boolean search queries, 104,316 resolved references, and structured methodological metadata. For controlled evaluation, the original Boolean queries are additionally provided in a normalized, approximated form operating over titles and abstracts. To illustrate the intended use of the collection, baseline experiments compare the approximated expert Boolean queries with zero-shot LLM-generated Boolean queries, BM25, and dense retrieval under a unified evaluation setting. The results highlight systematic differences in precision, recall, and ranking behavior across retrieval paradigms and expose limitations of naive zero-shot Boolean generation. SR4CS is released under an open license on Zenodo (https://doi.org/10.5281/zenodo.17163932), together with documentation and code (https://github.com/webis-de/scolia26-sr4cs), to enable reproducible evaluation and future research on scaling systematic review automation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。