让AI先理解行为规范再写代码,显著提升从零开始编程的准确率。
SpecFirst: Behavioral Specification Elicitation as a First-Class Step in Agent-Based Program Synthesis from Scratch
- 分两阶段:先由专用代理通过测试用例和文档生成结构化行为规范
- 在200个ProgramBench任务上,测试通过率提升6.9%~21.3%,探索覆盖率提高9.4%~18.5%
- 适合需要从零构建程序的AI开发场景,尤其对提示词理解不稳定的模型有效
基于大模型的智能体在已有代码库上下文中表现优异,但在完全从零构建程序时仍面临巨大挑战。现有基准如ProgramBench显示,仅凭自然语言文档和可执行二进制作为行为验证器,前沿模型解决率不足1%。当前框架将文档阅读、行为探索与代码生成合并为单次流程,导致探查不足、意图丢失及早期误解传播。受经典需求工程启发,本文提出SpecFirst——一个将行为规范获取作为独立首阶段的两阶段框架。首先,专用规范代理通过探测二进制并结合文档生成结构化规范;随后,代码合成代理基于该规范进行实现。该分解机制在编码前澄清文档歧义,并提供稳定的行为参考。我们在四个不同能力水平的模型(跨越两个模型家族)上对全部200个ProgramBench实例进行了评估。SpecFirst一致优于单循环基线,测试通过率提升6.9%-21.3%,二进制探索覆盖率提升9.4%-18.5%,均具统计显著性。进一步行为分析表明,预设规范使代码构建更早且持续。结果证明,显式的“需求工程”阶段是零起点程序构建的有效范式。
原文摘要 · Abstract (English)
LLM-based agents excel at software engineering tasks where an existing codebase provides context, but constructing a program from scratch remains fundamentally harder. Recent benchmarks such as ProgramBench quantify this gap: given only natural-language documentation and an execute-only binary as a behavioral oracle, even frontier models solve fewer than 1% of instances. Existing frameworks conflate documentation reading, behavioral exploration, and code synthesis into a single pass, causing agents to probe insufficiently, lose behavioral intent as context drifts, and propagate early misinterpretations into the final implementation. Inspired by classical requirements engineering, we argue that behavioral specification elicitation should be a first-class phase that precedes implementation. We present SpecFirst, a two-stage framework that forces the specification elicitation before code synthesis. A dedicated spec agent first probes the binary and combines observations with documentation into a structured specification. Next, a code synthesis agent then uses this specification to drive implementation. This decomposition resolves documentation ambiguities before coding begins and provides a stable behavioral reference throughout synthesis. We evaluate SpecFirst on all 200 ProgramBench instances across four models spanning two families and an order of magnitude of capability. SpecFirst consistently outperforms the single-loop baseline, improving test pass rates by 6.9%-21.3% and binary exploration coverage by 9.4%-18.5%, all statistically significant. Behavioral analysis on code synthesis further shows that a prior specification enables earlier and more sustained code construction. Our results demonstrate that an explicit requirements-engineering phase is an effective paradigm for from-scratch program construction.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。