测试代码智能体在模糊需求和多轮迭代中的表现
Asuka-Bench: Benchmarking Code Agents on Underspecified User Intent and Multi-Round Refinement
- 用多轮反馈闭环模拟真实开发流程
- 8个模型在50个任务中通过率差距达38个百分点
- 适合评估代码生成与纠错能力的科研人员
现有代码生成基准仅评估完整提示到单次输出的映射，但真实网页开发中用户常未明确需求，需通过查看中间结果逐步澄清。我们提出Asuka-Bench，一个基于浏览器渲染行为、结合模糊需求与多轮迭代反馈的基准。每个任务通过闭环流程解决：代码智能体生成项目，UI智能体在部署站点上执行测试，用户大模型将评估结果转化为自然语言反馈用于下一轮。基准包含50个网页任务，784项评估标准和2402个预期结果。我们在8个大模型上对比了两种代理框架，结果显示加权任务通过率差异达38个百分点，且各模型修复反馈的能力差异显著。该基准尚未饱和：最强模型在三轮内仅完成52%的项目。
原文摘要 · Abstract (English)
Existing code-generation benchmarks score a single mapping from a complete prompt to a one-shot output. However, real web development is different. Users seldom write a full spec at the start; many requirements only become clear once they look at an intermediate result and react to it. We present Asuka-Bench, a benchmark that pairs underspecified user intent with multi-round refinement, grounded in browser-rendered behavior. Each task is resolved through a closed loop: a Code Agent generates a web project, a UI Agent executes test cases on the deployed site, and a User LLM turns evaluation outcomes into natural-language feedback for the next round. The benchmark comprises 50 web tasks with 784 evaluation criteria and 2402 expected outcomes. We benchmark 8 LLMs across 2 agent frameworks. The results separate models clearly: weighted Task Pass Rate varies by 38 percentage points and models also differ substantially in their ability to repair from feedback. Asuka-Bench is also far from saturated: even the strongest model completes only 52% of projects after three rounds.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。