SPICE自动化标注工具让软件工程数据集制作成本降为人工的0.05%
SPICE: An Automated SWE-Bench Labeling Pipeline for Issue Clarity, Test Coverage, and Effort Estimation
- 用智能代码导航与多轮共识生成标注，替代人工
- 1000条数据标注成本从10万降至5.1美元
- 适合需要大规模训练数据的研究者
高质量标注数据对训练和评估软件工程领域的基础模型至关重要，但传统人工标注成本高、耗时长。本文提出SPICE，一种可扩展的自动化管道，用于标注类似SWE-bench的数据集，包含问题清晰度、测试覆盖率和工作量估算三类标签。SPICE结合上下文感知代码导航、基于推理的提示策略与多轮共识机制，生成接近专家标注的结果。该设计源于作者在标注超过800个SWE-Gym实例时的经验与挑战。SPICE在与人类标注的SWE-bench Verified数据对比中表现良好，将1000条数据的标注成本从约10万美元降至仅5.10美元。研究还发布了SPICE工具及SPICE Bench数据集，包含6802个经标注的实例，源自291个开源项目，规模超SWE-bench Verified的13倍。
原文摘要 · Abstract (English)
High-quality labeled datasets are crucial for training and evaluating foundation models in software engineering, but creating them is often prohibitively expensive and labor-intensive. We introduce SPICE, a scalable, automated pipeline for labeling SWE-bench-style datasets with annotations for issue clarity, test coverage, and effort estimation. SPICE combines context-aware code navigation, rationale-driven prompting, and multi-pass consensus to produce labels that closely approximate expert annotations. SPICE's design was informed by our own experience and frustration in labeling more than 800 instances from SWE-Gym. SPICE achieves strong agreement with human-labeled SWE-bench Verified data while reducing the cost of labeling 1,000 instances from around \$100,000 (manual annotation) to just \$5.10. These results demonstrate SPICE's potential to enable cost-effective, large-scale dataset creation for SE-focused FMs. To support the community, we release both SPICE tool and SPICE Bench, a new dataset of 6,802 SPICE-labeled instances curated from 291 open-source projects in SWE-Gym (over 13x larger than SWE-bench Verified).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。