评估大模型代理生成微服务应用的能力与难点
LLM Agents for Generating Microservice-based Applications: how complex is your specification?
- 设计标准规范模板与难度评分机制
- 强模型在中等难度下表现良好，高难度时失败率高
- 细粒度生成可提升代码正确性，适合开发者参考
本文评估大模型代理在生成真实世界问题代码方面的能力，聚焦于微服务架构应用的代码合成。我们定义了应用规格的标准模板，并提出一种评分指标来衡量规格复杂度：分数越高，代码生成难度越大。实验结果表明，使用强大LLM（如GPT-3o-mini）的代理在中等复杂度规格上表现尚可，但在高复杂度场景下表现不佳，原因包括更复杂的业务逻辑、更多外部服务调用、数据库集成及认证等非功能需求。我们分析了生成代码中的错误，揭示了大模型代理在处理复杂规格时的关键挑战。最后，通过采用细粒度生成策略，显著提升了生成代码的正确性。
原文摘要 · Abstract (English)
In this paper we evaluate the capabilities of LLM Agents in generating code for real-world problems. Specifically, we explore code synthesis for microservice-based applications, a widely used architectural pattern for building applications. We define a standard template for specifying these applications, and we propose a metric for scoring the difficulty of a specification. The higher the score, the more difficult it is to generate code for the specification. Our experimental results show that agents using strong LLMs (like GPT-3o-mini) do fairly well on medium difficulty specifications but do poorly on those of higher difficulty levels. This is due to more intricate business logic, a greater use of external services, database integration and inclusion of non-functional capabilities such as authentication. We analyzed the errors in LLM-synthesized code and report on the key challenges LLM Agents face in generating code for these specifications. Finally, we show that using a fine-grained approach to code generation improves the correctness of the generated code.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。