用AI自动生成并验证Python代码漏洞，发现多个真实缺陷。
Agentic Property-Based Testing: Finding Bugs Across the Python Ecosystem
- 用大模型分析代码和文档，自动合成性质测试并执行
- 在100个热门包中发现56%为有效漏洞，Top21中86%有效
- 适合开发者、开源维护者用于自动化质量检测
属性测试（PBT）是一种轻量级形式化方法，通常以随机测试框架实现。用户通过框架提供的组合子定义输入域，以单元测试函数表达预期性质或不变性，框架则通过生成输入并调用测试函数来寻找反例。本文展示了一个基于大语言模型（LLM）的智能体，能分析Python模块，从代码与文档中推断函数级及跨函数性质，自动生成并执行PBT，反思测试结果以确认真实漏洞，并输出可操作的漏洞报告。我们在100个流行Python包上进行了广泛评估，经人工审查，该智能体生成的报告中56%为有效漏洞，其中32%我们愿意提交给维护者。通过构建排序规则筛选高优先级漏洞，前21个中86%为有效漏洞，81%可提交。这些漏洞涵盖序列化失败、数值精度错误、缓存实现缺陷等多种类型。我们已报告5个漏洞，4个附带补丁，包括对NumPy和云计算SDK的修复，其中3个补丁已被合并。结果表明，结合大模型与属性测试可实现严谨且可扩展的自主软件测试。代码与数据集已公开：https://github.com/mmaaz-git/agentic-pbt。
原文摘要 · Abstract (English)
Property-based testing (PBT) is a lightweight formal method, typically implemented as a randomized testing framework. Users specify the input domain for their test using combinators supplied by the PBT framework, and the expected properties or invariants as a unit-test function. The framework then searches for a counterexample, e.g. by generating inputs and calling the test function. In this work, we demonstrate an LLM-based agent which analyzes Python modules, infers function-specific and cross-function properties from code and documentation, synthesizes and executes PBTs, reflects on outputs of these tests to confirm true bugs, and finally outputs actionable bug reports for the developer. We perform an extensive evaluation of our agent across 100 popular Python packages. Of the bug reports generated by the agent, we found after manual review that 56\% were valid bugs and 32\% were valid bugs that we would report to maintainers. We then developed a ranking rubric to surface high-priority valid bugs to developers, and found that of the 21 top-scoring bugs, 86\% were valid and 81\% we would report. The bugs span diverse failure modes from serialization failures to numerical precision errors to flawed cache implementations. We reported 5 bugs, 4 with patches, including to NumPy and cloud computing SDKs, with 3 patches merged successfully. Our results suggest that LLMs with PBT provides a rigorous and scalable method for autonomously testing software. Our code and artifacts are available at: https://github.com/mmaaz-git/agentic-pbt.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。