arXiv:2506.10998cs.SEcs.AI2025-06被引 1

用大模型自动验证后端代码逻辑,一半测试工作可被替代。

Towards Automated Formal Verification of Backend Systems with LLMs

  • 将Scala代码转为形式化数学表达,自动生成可验证的规范
  • 50%以上测试需求可被自动证明正确,失败则暴露缺陷
  • 每接口仅需2.19美元,适合大规模自动化测试场景

软件测试对保障系统行为正确至关重要。现有自动化测试方法因测试局部性、可靠性不足和业务逻辑盲区,难以媲美人类工程师。本文提出新框架,利用函数式编程与类型系统,将Scala后端代码转换为形式化Lean表示。管道自动生成描述API与数据库操作预期行为的定理,并使用基于大模型的证明器进行验证。若定理被证明,则对应逻辑必然正确,无需再测试;若其否定被证明,则确认存在缺陷。当两者均无法证明时,需人工介入。我们在真实后端系统上评估,发现该方法可形式化验证超过50%的测试需求,表明一半测试工程师的工作量可被自动化。此外,平均每个API成本仅2.19美元,远低于人工测试,且可通过并行执行轻松扩展。结果表明,该方向在可扩展的AI驱动测试中具有广阔前景,随着模型进步,有望大幅提升工程效率。

原文摘要 · Abstract (English)

Software testing plays a critical role in ensuring that systems behave as intended. However, existing automated testing approaches struggle to match the capabilities of human engineers due to key limitations such as test locality, lack of general reliability, and business logic blindness. In this work, we propose a novel framework that leverages functional programming and type systems to translate Scala backend code into formal Lean representations. Our pipeline automatically generates theorems that specify the intended behavior of APIs and database operations, and uses LLM-based provers to verify them. When a theorem is proved, the corresponding logic is guaranteed to be correct and no further testing is needed. If the negation of a theorem is proved instead, it confirms a bug. In cases where neither can be proved, human intervention is required. We evaluate our method on realistic backend systems and find that it can formally verify over 50% of the test requirements, which suggests that half of a testing engineer's workload can be automated. Additionally, with an average cost of only $2.19 per API, LLM-based verification is significantly more cost-effective than manual testing and can be scaled easily through parallel execution. Our results indicate a promising direction for scalable, AI-powered software testing, with the potential to greatly improve engineering productivity as models continue to advance.

形式化验证大模型自动化测试后端系统

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。