用真实开发场景自动构建可复现的编程助手评估基准。
REAP: Automatic Curation of Coding Agent Benchmarks from Interactive Production Usage
- 从真实开发会话中自动提取高质量任务，无需人工标注。
- 通过多轮测试与LLM验证，确保评估结果可靠且可复现。
- 适合关注生产级代码生成模型落地的开发者与研究者。
AI编程助手在生产环境部署需快速、可复现的评估信号。现有工业实践在速度与保真度间权衡：线上A/B测试耗时数周且影响用户体验，影子部署结果不可复现，公开基准则与生产工作负载在语言分布、提示风格和代码库结构上存在偏差。本文提出REAP（Relevance and Execution-Audited Pipeline），一个自动化基准构建流水线，从真实开发者-代理会话中无监督地构建生产相关基准。该流程面临不可测提示、测试错位和测试波动等挑战。虽可通过人工审核筛选高质量任务，但在大型单体仓库（monorepo）中不可持续——因构建环境状态瞬变，需持续对最新代码库重新校准。为此，REAP引入基于LLM的任务分类、代理式测试相关性验证及多轮稳定性检测，保障可执行基准输出可信信号。我们使用REAP构建了Harvest基准，每个任务以真实开发者提示输入编码代理，并通过生产环境中获取的失败测试用例验证生成代码变更。Harvest覆盖超过四种编程语言，多数任务源自Hack。模型与框架评估显示，五款前沿模型解决率在42.9%至58.2%之间，揭示出显著能力差异，为实际部署提供决策依据。
原文摘要 · Abstract (English)
Production deployment of AI coding agents requires fast, reproducible evaluation signals. Existing industrial practices trade off speed and fidelity: online A/B testing takes weeks and risks user experience, shadow deployment yields signals that are not reproducible across runs, and public benchmarks diverge from production workloads in language distribution, prompt style, and codebase structure. This paper presents REAP (Relevance and Execution-Audited Pipeline), an automated curation pipeline that constructs production-derived benchmarks from real developer-agent sessions without manual labeling. Such curation, while in-distribution to production usage, runs into several challenges. Untestable prompts, misaligned tests, and test flakiness all compromise evaluation reliability. While tasks can be manually audited to ensure only high-quality tasks remain in the benchmark, this approach is infeasible in the monorepo setting: the build infrastructure state is often ephemeral in large monorepos and requires the benchmark to be continuously re-curated against the current codebase. As manual verification cannot be sustained at this cadence, REAP adds an automated verification layer using LLM-based task classification, agentic test-relevance validation, and multi-run stability checks to ensure the executable benchmark yields trustworthy signals. We use REAP to curate Harvest, a benchmark where each task feeds the coding agent a real developer prompt and verifies the resulting code change against fail-to-pass tests retrieved from production. Harvest's distribution spans more than four programming languages with a majority of tasks drawn from Hack. Model and harness evaluations reveal that solve rates range from 42.9% to 58.2% across five frontier models, surfacing capability differences that inform concrete deployment decisions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。